Apple and fingerprints

Direct link to the last info

July 2012: Apple acquires Authentec for $356 Million

What a surprise! So the big firm from Cupertino is now interested into biometrics? During years, competition has tried to get the killing application using biometrics -just have a look at my website- with some mitigated success.

And Apple has certainly some plans, because buying the first fingerprint sensor manufacturer able to produce millions of pieces, and the only real one since the merge between the number one and number two, Authentec and Upek, is not a small step towards biometrics.

As expected from Apple, access to the Authentec fingerprint sensors has been reduced, not saying closed but almost, which is a difficulty for all companies that were using their sensors, and particularly Apple's competition. So it is somewhere a good news for small fingerprint sensor manufacturers: there is an opportunity to enter this market!

But there is a big mystery, as usual with Apple, about how a fingerprint sensor will be included in their products. So speculations and leaks started... As usual, the only reliable source of information are the patent applications, but you never know if it will be used, and when. But that's better than nothing.


Note: if you don't want to read the original patent applications (something I understand :) you will find also some analysis at Patently Apple, a pretty good website.

I give very few comments hereafter, for sure, there is much more to say about all those patent applications... but this is nearly a fulltime job to do so.


Oct, 2012: Patent application : access to internal devices

The 20120258773 application from Alvarez, Dinh & Myers discloses a device with a window relealing a biometric sensor (or other things).

The image speaks for itself.

application 20120258773

April, 2013: Production of the iPhone 5S 'set to be delayed beyond June because of its fingerprint sensor'

The MailOnline article says that:

  • Suppliers say coating material for the sensor is delaying manufacturing
  • Believed sensor could be used to unlock the handset and replace passwords

So there is a fingerprint sensor! Making the design of a new chip is long and difficult: the chip has to work, and also ramping up the production to reach millions of pieces is pretty difficult. And moreover, there will be a packaging problem, I know that the coating is not an easy thing to handle. It makes sense...


June, 2013: Patent application : integrally molded die and bezel structure...

The 20130154031 application from Bond, Kramer (Upek), Gozzini unveils a bezel to encapsulate a swipe fingerprint sensor.

This is making sense, as it is required to protect the sensor, as well as the wired connections on the side.

application 20130154031

July 16, 2013: Apple reportedly having some IC yield problems with new iPhone manufacturing

The DigiTimes article says that:

Second warning, after the April info. TSMC was the usual foundry of Authentec, and having yield problems means this is a new chip.


July, 2013: Patent application : finger sensor, coupling electrodes

The 20130181949 application from Dale Setlak (Authentec R&D).

In a few words, this is about integrating a (more or less) capacitive swipe fingerprint sensor in a touchscreen display. So we learn that it is only a small part of the screen that is used, and it is likely pretty difficult to integrate with the regular capacitive sensor that goes with the touchscreen. Definitively something long to develop, it would be a surprise to be able to integrate this with a product in a close future...

Also, integrating a swipe sensor directly with the touchscreen, where you have to slide to unlock, would be very convenient. But hard to make from a technological point of view.

BTW, you will see that this application is starting with a reference to a patent from Dale Setlak, immediatly followed by the U.S. Pat. No. 6,289,114, from ... myself (and dated September 11, 2001 which is weird).

application 20130181949

August, 2013: Patent application: differential measurements

The 20120085822 application from Setlak, Landy, Straub, Schenk is about a sensor making differential measurements.

This patent application is interesting mainly because it adresses the problem of the protective coating thickness above the pixels, a very hard problem to solve. If you want to have a nice external appearance of the sensor, and some protection, it is required to add something above the pixels. But doing this, the resulting signal is decreasing, leading to faint images, or even no image at all if it is too thick. And this is a critical problem for Apple.

application_20120085822_coating
application_20120085822_fingerprint_differential

Another interesting fact is this differential approach, which gives images that looks like 3D, a typical feature. We got the same kind of 3D images when using the FingerChip, which was a pure differential device as measuring temperature changes in time (not spatially). (To my former collegues that may see that: this is not a FingerChip image, I have been surprised myself also about similarities.)


Sept. 5, 2013: leaks about the iPhone 5S internal

Sonny Dickson unveils some Behold The First Part Leaks of the Grey or ‘Graphite’ iPhone 5S.

So we can see the internal, and maybe the fingerprint sensor? Unfortunaly, there is nothing. There is no hole in the front panel for a sensor, and the button seems to have nothing special. Maybe it is not a 5S, or even this is a version without fingerprint sensor.


Sept. 5, 2013: Patent application : Electronic Device With Shared Near Field Communications and Sensor Structures

Hidden behind an unclear name, the 20130231046 application from Pope, Jarvis, Merz & Myers unveils the shape of the future home button with a fingerprint sensor.

A few days after, it is obvious, see below.

application_20130231046_home_button

Sept. 5, 2013: leaks about the fingeprint scanner

Sonny Dickson unveils some High-Res Images of the iPhone 5S Fingerprint Scanners.

Sorry guy, there is no sensor, but the square area is intriguing, something is obviously missing. And with flexible circuits, you never know how it is installed in the final device. This circuit is making more sense than what has been unveiled on August, 25, and is not the same...


Sept. 10, 2013 : the iPhone 5S contains a fingerprint sensor

The wait is over.

Apple unveils its own version:

Let's start with the official statements:

... packed into its beautiful and refined design are breakthrough features that really matter to people, like Touch ID, a simple and secure way to unlock your phone with just a touch of your finger.

iPhone 5s introduces Touch ID, an innovative way to simply and securely unlock your iPhone with just the touch of a finger. Built into the home button, Touch ID uses a laser cut sapphire crystal, together with the capacitive touch sensor, to take a high-resolution image of your fingerprint and intelligently analyze it to provide accurate readings from any angle. Setting up Touch ID to recognize your fingerprint is easy, and every time you use it, it gets better. The Touch ID sensor recognizes the touch of a finger so the sensor is only activated when needed, preserving battery life. All fingerprint information is encrypted and stored securely in the Secure Enclave inside the A7 chip on the iPhone 5s; it’s never stored on Apple servers or backed up to iCloud®.

What's important:

There is a little more technical information during the presentation from Phil Schiller (senior VP of worldwide marketing, Apple) (click on image to enlarge):

500 dpi: nothing special, this is the usual numbers we knows for years. One pixel is 50 microns.

170micron thick: OK, the chip is thinned, this is because they have not very much space, but this is pretty regular in semiconductor industry.

Reading the "subdermal layer" : this is the usual speech about Authentec sensors. Ah! marketing guys...

"The stainless ring is used as a detector" : OK, this makes sense. But the usual Authentec technology is using a ring to send an electrical signal in the skin, so that the infamous subdermal layer can participate to the reading of the fingerprint: is it still the case, or do they move to a pure capacitive sensor? I would guess no, because they added this sapphire layer which is not a good idea to read fingerprint -the distance between the sensor and the skin increases-, but is a brillant idea to get a nice design. Let's try to get more clues.

The official video gives some more informations:

Opps! there is one more layer? the green one (changing color later in the video...)? That is curious, because adding layers between the skin and the pixels is NOT a good idea, as it reduces the collected signal because of the increased thickness, and you may get some trouble with the air caught between the layers... We don't know if the silicon sensor is really like that, but at least we are sure it is a square sensor. Also, obviously, the button must be flat because the silicon is flat, and to have an homogeneous image, it's better not having a concave area. And it helps the manufacturing.

Maybe it is only some glue layer, so that there is no air caught between the silicon and sapphire. The high relative dielectric permittivity of sapphire (~10 where air is 1) will help a lot the capacitive sensor. Apple should have made their sensor using SOS, silicon on sapphire, with the sensing area on the backside, so the wiring is no more a problem... (I made my first circuit using SOS).

Or just a mistake of the marketing guys :)

Ergonomics

The first image below is scaring for someone used to work with fingerprints:

The side of your thumb is certainly not the best place to get a usable fingerprint. Moreover, it is likely that the captured skin area is reduced, lowering the recognition performances. The second image is a much more better situation. At least, enrollment has to be special, because it is required to collect a lot different images. And people must learn that if you enroll like the second image, the first image will not be recognized. It is not clear if the stored template is enhanced AFTER the initial enrollment. That would help the recognition performances, but this may be also dangerous for the false acceptance rate...

Apple is making some efforts to explain how it is working, so we know that it is a minutia-based algorithm, which makes sense with a 500 dpi sensor. The fingerprint logo is nice, but well, anyone working with fingerprints knows that the lines are not broken as shown on the logo. And there is NO bifurcation (excepts the delta, sure), which never happens.

Not showing the real fingerprint image is a good thing from a security point of view. It is not a good thing for someone like me that would like to have an idea of the quality of the sensor, which is questionable because of the added layers over the silicon sensor.

What about false acceptance, false rejection, spoofing?

The whole biometric community hopes that the fingerprint sensor of the iPhone 5S will properly work, because Apple devices are so exposed, if something wrong happens, then it could close the consumer market to biometrics...

What could be a nightmare?

  1. Not recognising the owner (convenience), that is having a too high false rejection rate
  2. Making mistakes between persons (security), that is having a too high false acceptance rate
  3. Being spoofed, that is a fake finger is recognized as a true finger
  4. Having security trouble related to the software and/or template storage

Apple is not disclosing anything about false accepts, false rejects: the marketing guys have chosen to act as if it was obvious, I suppose, so that users will be confident with the technology. Fingerprint recognition never fails.

  1. False rejection / convenience
  2. Looking at the first videos "hands on the iPhone 5S", it looks like it is working fine, and that's a minimum for a device coming from Apple where convenience is a must. The pretty long enrollment sequence where the user is asked to touch the sensor several times surely helps. Moreover, it is easy to adjust the internal threshold so that it works reasonably well. I didn't see any adjustment available to the user up to now, something that helps to adjust the security threshold, so I believe that it is an arbitrary marketing choice not bothering the user with such adjustment. And maybe the threshold is automatically adjusted, depending on the image quality linked to the fingerprint of this particular user.

    We don't know if the system continuously learns, that is each time a positive identification is done, then the template is enhanced with the new data.

    About what is commonly called "bad fingers", I expect that the sensor will be sensitive to dry fingers, chalk, moist fingers, mainly because of the thickness of the sapphire coating that does not help. But we may have a good surprise if what is described in the patents is working well, you never know.


  3. False accept / security
  4. What we don't know is the probability to get a false acceptance.

    Is it 1%? is it 0.01%? In other words, how many non-enrolled fingerprints should I try to enter the system, from a statistical point of view? At least, there is a clue with the sensor size. Obviously, if the sensor is too small, then it will be easily confused as you can expect. On the other hand, if the sensor is larger than the fingerprint, then it is the best we can do.

    Silicon cost is directly linked to the area of silicon. This is why I invented the silicon swipe sensing in the past. The main question is: what is the minimum size to get reasonable results? Then, we will try to know the size of the touchID sensor.

    In 2004, I published a paper with Cogent "Reducing Silicon Fingerprint Sensor Area" at the International Conference on Biometric Authentication ICBA 2004 Hong-Kong July, 15-17 2004
    Lecture Notes in Computer Science, Volume 3072 / 2004, pp. 301 - 308

    In this paper, we concluded that, with the best algorithm available (Cogent is making fingerprint systems for the police), there was clearly a limit with a 7mm overlap, AND we were speaking of swipe sensor, that is the height was much more. In other words, if you make a sensor using minutiae (as explained by Apple) that is less than 7x7mm, the performance results will be bad. If you want to make better, then you must go to a better resolution to use the third level properties (the pores).

    Another way to get an idea of this minimum is the size of devices that have been done in the past, just have a look at this list. We see that all sensor manufacturers did not go below 6.5 mm (conveniently 128 pixels) for swipe sensors, and only Authentec made a 6.5 x 6.5 mm sensor in the past, and it was not the one that gave the best results.

    Just to give an idea, the square on the image below shows a 7x7 mm area. Do the matching!

    matching_card_latent_square77

    So what is the size of the touchID sensor? It is, at best, the square inside the stainless steel ring. Looking at the nice images from the Apple website, and knowing the width of the device (58.6mm), I estimate the internal size of the ring to 10mm, so containing a 7x7mm square. Considering that making a 128x128 pixel sensor is practical, my guess is that the touchID sensor is a 6.4x6.4mm square for the sensing area.

    (A few days after, plans of the iPhone 5S were released (a short time...), and the ring is 10.9mm, external)

    We are at the limit. So my guess is that we will quickly see some false accept reports.


  5. Spoofing
  6. Concerning spoofing, it seems that we keep the old marketing way from Authentec, that is: the sensor is reading the subdermal layer of the skin. So the average person will think that a fake cannot work, as the sensor is reading the subdermal layer. But saying that you read the subdermal layer (which is certainly true in the case of the skin) does not mean that other modes are not possible!

    At least, there is nothing specific said about aliveness recognition, no specific sensor. IF it is exactly the same technology than Authentec used to made, and it is likely given the short time to create and produce a new sensor, then we'll have the same results about spoofing (and I'm used to spoof fingerprint sensors).

    So it is just a matter of waiting till the first hacker will release a video with a fake finger accessing the iPhone 5S. It should be easy, considering that it is a static sensor (spoofing a swipe sensor is more complicated), and that the enrolment sequence will help to see if the sensor acquires something or not...


  7. Electronic security
  8. About all this security side related to cryptography, storage of the templates... there is no information about all this stuff, and obviously, there is nothing else to do at the moment than naively believe what Apple is saying about this. If Apple has properly done the work, then everything is hidden in the processor and never gets out. Proving that is difficult without opening the internal. But if it is not hidden, some leaks will happen.


Now, I'm waiting for more information, once the iPhone 5S will be opened so we can see the silicon. Or the first reported false accept or spoof...


Sept. 13, 2013: Jefferies' Peter Misek Says 'Terrible Yields' On iPhone Fingerprint Sensor Hurting Production

Forbes article

One more rumour about that, after April and July.


Sept. 19, 2013: Watch A Cat Unlock The iPhone 5s Using Touch ID And The Fingerprint Sensor

TechCrunch article / Darrell Etherington shows that a cat's paw is able to unlock the iPhone 5S. And also some skin areas where fingerprints are not so usable.

So Apple's explanation about fingerprint minutiae is not the only way to recognize. Now we know that some form of pattern recognition is also used, or the cat -without fingerprints- should not work, and even cannot be enrolled.

As a result, it will not be very long till a false accept will be reported. Making pattern recognition over a so small area cannot be very accurate. Especially if you store many small areas in the template, you increase the chance of true recognition, but also false accepts...

The day after, it is a dog... iPhone 5S Fingerprint sensor works for dogs too! CNN iReport.


Sept 20, 2013: more info about Touch ID on Apple website

Apple disclose on their website (support) some information related to false accepts:

That's interesting. But it creates much more questions that requires answers...

  1. 1 to 50,000 probability applies to ... what?
  2. Is this the probability every time I touch the sensor? It is likely this situation, because it gives better numbers. Remember that you are allowed to touch several times, with different areas of your skin, so each try increases your chances to enter... and you can enroll several fingers, so testing against several templates at the same time.

    It is possible to enroll 5 fingers, so the final probability is ... 1 in 10,000. And it is limited to 5 attempts, so the probability falls to 1 in 2000.

    Finally, comparison with the 4 digit pincode is not good, as described in this page: Myth: A FAR of 10-4 = a 4 digit PIN code. The situation is much more better in the case of the fingerprint, and Apple could take advantage of this in their description... Maybe Apple considered that it was too complicated for their average user.

  3. 1 to 50,000 probability: how did you get this number?
  4. Well, this is a nice FAR (false accept rate) for such small sensor. There are many biometric modalities that would like to have this kind of performances, together with the good FRR (false reject rate) as observed, likely a few % -there is (not yet) reports about fingers that were not able to be recognized, and it will not be the case as the sensor is already working for difficult prints such as cats and dogs...

    Did Apple tried with 50,000 persons, well even 5,000 with ten fingers ? I don't think so, because it is a too long work (and how keeping such secret?), and you cannot rely on only one test to get some probabilities. In general, there is some data collection (collecting raw images), and the recognition algorithm is tested against the database, and you get the raw FRR and FAR. Then, you need to translate this into your system, here it seems that several images are taken and stored, looking at the enrolment procedure. It is also important to acquire your database using a system representative of your final system, especially with the same ergonomics. Well, the basics for any biometric professional guy...

    As a reminder, the myth of the 0.0001% FAR and 0.0001% FRR system. Here, it is 0.05% FAR, FRR unknown, likely 2%, maybe even 10%.


Sept. 20, 2013: iPhone 5s teardown

Here is the home button (credit: Ifixit).

Hey guys, remove the sapphire, so we can see the silicon sensor and estimate the thickness! But it is likely very well glued...


Sept. 21, 2013: iPhone 5S fingerprint sensor loophole or feature?

Note Suwanchote has posted a video on YouTube, showing it was possible to enroll and recognize 5 different fingers in one slot, so you can enroll up to 25 fingers.

This is a very interesting fact when connected with the 1 to 50,000 probability discussed just above, because this is lowering the announced FAR to 1 to 400 when enrolling 25 fingers and trying 5 times. We are not very far to see the first false accept...

This is also giving another important clue during enrolment: the algorithm does not search to create a larger print from the small raw images, which would be very difficult to make as the recovery would be too low to

Why is it working like that? Just to make sure that the owner of the iPhone is not rejected. Everything has been done to reduce the false rejection rate, because this is the very first feature the regular user sees. The more images you store, the better the recognition will be. Unfortunaly, if you relax too much the system so the good guys are not rejected, this is also the same for the bad guys, and the false reject rate increases.


Sept. 21, 2013: Chaos Computer Club breaks Apple TouchID

The Chaos Computer Club, which is used to spoof fingerprint sensors, has fooled the touchID sensor very quickly, with some usual fake finger (see the video on YouTube), even if they did not understand how it happened from a physical point of view (it is only a matter of electrical conductivity with the stainless steel ring, this is why they need moisture...) and said something wrong about the resolution of the sensor which is 500 dpi, as all sensors produced by Authentec excepted the oldest ones, 10 years ago.

There are simpler ways to do the trick, but well, no matter, it tooks less than two days to spoof it. So there is nothing special in this sensor related to aliveness detection, it is working as usual.

  • (Sept 23) Another hack: Why I Hacked Apple’s TouchID, And Still Think It Is Awesome

  • Oct. 28, 2013: Chipworks get a look at the fingerprint sensor chip

    Chipworks iPhone 5s Touch ID – Technology Perspective article describes the silicon chip: they teared down a button.

    Stacking

    Finally, the stack of materials described in the presentation movie was pretty good! We can see this green-blue layer which is a kind of glue between the sapphire and the chip. Unfortunately, Chipworks didn't say a word about the thickness, which is a big problem concerning the resolution as it blurs the images: imagine you have one pixel every 0.05mm, and what is above is more than 0.10mm, twice the pitch!

    Wire bonding

    Concerning the chip itself, Chipworks has an excellent note: An unusual feature of the die is the dark areas at the top and bottom edges. It appears that the silicon has been partially etched away to provide a recessed shelf within the die area for the wire bonds. That allows the top surface of the chip to be bonded straight onto the sapphire disc, minimizing the finger-to-chip distance. That's extremely unusual. Another technique would have to use crossing vias (silicon-thru) to get the bondpads on the back of the chip, but this is likely not yet available -or too expensive- in the foundry they used.

    Sensing area size

    The chip is 6.6mm x 6.1mm (I guessed 7x7mm), with an array of 88x88=7744 pixels at 500dpi (so the usual 50.8 microns pitch of fingerprint sensors), so the sensing area is as small as 4.5mm x 4.5mm: that's too small to get good recognition performances! I was expecting 128x128 pixels (the purple square), and was not confident, so with 88x88 pixels (the red square), and moreover with a thick layer that blurs images, forget it...

    This is mainly described in the 20130307818 application from Pope & als.


    Nov. 4, 2013: GT Advanced Technologies Inc. Signs Multi-Year Sapphire Materials Agreement With Apple

    GT Advanced Technologies Inc. press release.

    So we can guess that Apple will merge the sensor and the display one of these days...


    Sep. 2015: A new sensor chip, with a larger sensing area.

    Not a real surprise: increasing the sensing area while keeping the same button will help to enhance the recognition rates.

  • (2015 Sep) a new sensor chip + NVT8417 companion chip (dated 2014).
    The resolution seems to be still 508 dpi, but looks like it is not a square chip: 112x96 pixels (I am unsure if I'm still able to properly count pixels, that's weird numbers...), so a little bit better than the previous version.
    They are now using TSV to avoid the recess and ease the packaging.

  • Jan. 2016: Rumours about a display with integrated fingerprint sensor.

    So the rumours about the next iPhone 7 are starting :)

  • (2016 Jan) The taiwanese site www.apple.club.tw is showing a photo of a display with some connectors. It's enough to speculate about a integrated fingerprint sensor, something hard to believe at the moment (it is *very* complicated to produce, look at Qualcomm with its new ultrasonic sensor, which is installed... on the back of the phone!). I would better believe that the sensor is beside the display, maybe sharing the same sapphire surface -something still complicated to manufacture.
  • (2017 Feb) Apple granted patent No. 9,582,102 for fingerprint recognition using entire iPhone display. Seems not to be the most innovative patent, considering some other existing sensors, for instance Trais.

  • Sep. 2016: a new minor version of the sensor.

  • (2016 Sep) The companion chip is the NVT8418-A0 dated 2015. Looks to be just an update, no change in the chip size.

  • Oct. 2016: Apple patents fingerprint sensor that works through displays.

  • (2016 Oct) The tech described in Apple's U.S. Patent No. 9,460,332 for a "Capacitive fingerprint sensor including an electrostatic lens" may offers a solution to get the sensor behind the screen glass.

  • Feb. 2017: Acoustic Imaging System.

  • (2017 Feb) Apple introduces us to the Possible Next Generation of Touch ID for iDevices and Far Beyond
  • A patent about sound? Why not! Not sure we will see that in a close future. There are already some ultrasonic sensors, Qualcomm for instance.


    Feb. 2017: The iPhone 8 will feature optical fingerprint recognition.

    So the rumours about the next iPhone 8 are starting, as usual.

  • (2017 Feb) optical fingerprint sensor? source
  • (2017 Feb) Next iPhone may have facial recognition instead of a fingerprint reader
  • (2017 Feb) Taiwan Electronics Times reported: Home key integrated in the screen, no Synaptics, no Qualcomm, but a built-in sensor device. OLED screen adjusted because of fingerprint ID design.
  • (2017 Apr) Rumor: Integrating Touch ID into 'iPhone 8' display remains Apple's 'biggest bottleneck'. For the 5.8-inch OLED version, the biggest bottleneck remains integrating an under-glass fingerprint sensor into the display— the current yield rate of Apple's in-house AuthenTec solution remains low and AAPL seems unwilling to use other vendors' products Arcuri said.
  • (2017 Apr) iPhone 8's new fingerprint sensor may be on the fritz. Pacific Crest Securities analysts said in a note to investors Sunday that Apple's component suppliers were struggling with a new fingerprint sensor. That sensor is expected to go under the phone's cover glass to let Apple remove the iPhone's current home button and create a larger display.
  • (2017 Apr 19) Leaked iPhone 8 Schematic Has Touch ID Fingerprint Sensor Moved to the Back.
  • (2017 May 16) Apple awarded key iPhone 8 patents for in-screen Touch ID, edge-to-edge display and 3D scanner. This is not really new.
  • (2017 May 18) Leaked images show no fingerprint sensor: an In-Display fingerprint sensor? Likely not ready...
  • (2017 May 26) Apple has successfully finalized a solution to integrate Touch ID fingerprint recognition directly into the display of its upcoming 'iPhone 8', according to a new report on Friday,' Macrumours claims.
    Among several design changes TSMC reportedly discussed at the TSMC 2017 NA Technology Symposium was the lack of a home button on the redesigned OLED iPhone, owing to Apple's use of 'an optical fingerprint sensor to enable authentication directly on the screen' in the absence of a physical Home button.
  • (2017 June) Fingerprint sensor iPhone 8 while in the Power button? as suggested in a patent drawing? It could be a backup solution, Sony already done that in the past.
  • (2017 June 16) It seems that Vivo will propose a fingerprint sensor embedded in the screen. Before Apple. Grin!
    But which sensor?
    - The Synaptics optical sensor? May disturb the display: you may "see" where it is, because you need to have the light going to the sensor.
    - An ultrasonic sensor? Not so easy to glue below the screen, without destroying the pixels...
  • Wait & see!




    Comic written by Larry Lambert; illustrated by Jerry King.