Fish skeletons made of cartilage are rare in the fossil record because cartilage rarely fossilizes. However, the mineralized teeth and scales of these fishes can be abundant as fossils. Shark teeth are made of dentin laid down around the pulp cavity with a thin, durable outer layer of enameloid (enamel-like biomineral). Possible shark teeth have been found in fossil beds dating to the Ordovician (about 450 million years ago). Complete specimens of early sharks are first known from about 400 million-year-old rocks (the Devonian). Most of these ancient sharks died out at the end of the Permian (about 250 million years ago). New groups of sharks diversified during the Mesozoic, along with dinosaurs. These neoselachians (“new sharks”) included the ancestors of modern sharks. They ate cephalopods, fish, and perhaps even reptiles living in the sea. The body plan of modern sharks, including powerful jaws and a streamlined body propelled by a powerful tail fin, has allowed them to diversify and persist to the present day.

The earliest evidence of bony fishes is from fossils found in China dated to the early Cambrian (about 530 million years ago). The fossils are fragments of the head and gill covering of jawless fish (Agnatha), of which a few species persist today. The jawless bony fishes are not only the earliest bony fishes, but also among the earliest known animals with backbones (vertebrates) on Earth. Lacking jaws for chewing, they were filter feeders, straining small organisms from the water. By the early Devonian (about 400 million years ago), various groups of bony fishes with jaws had arisen. The earliest group of jawed fishes was the small, freshwater Ancanthodii, but they went extinct less than 150 million years later. Today's wildly successful group, the ray-finned bony fishes (Actinopterygii) began to diversify during the Devonian. Their earliest representatives had heavy scales, large eyes, and wide mouths.