{"id":284906,"date":"2014-03-17T09:00:06","date_gmt":"2014-03-17T16:00:06","guid":{"rendered":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/?p=284906"},"modified":"2016-09-06T10:21:45","modified_gmt":"2016-09-06T17:21:45","slug":"office-lens-snap","status":"publish","type":"post","link":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/blog\/office-lens-snap\/","title":{"rendered":"Office Lens Is a Snap"},"content":{"rendered":"

The moment mobile-phone manufacturers added cameras to their devices, they stopped being just mobile phones. Not only have lightweight phone cameras made casual photography easy and spontaneous, they also have changed the way we record our lives. Now, with help from Microsoft Research, the Office team is out to change how we document our lives in another way\u2014with the Office Lens (opens in new tab)<\/span><\/a> app for Windows Phone 8<\/a>.<\/p>\n

Office Lens, now available in the Windows Phone Store<\/a>, is one of the first apps to use the new OneNote Service API (opens in new tab)<\/span><\/a>. The app is simple to use: Snap a photo of a document or a whiteboard, and upload it to OneNote, which stores the image in the cloud. If there is text in the uploaded image, OneNote\u2019s cloud-based optical character-recognition (OCR) software turns it into editable, searchable text. Office Lens is like having a scanner in your back pocket. You can take photos of recipes, business cards, or even a whiteboard, and Office Lens will enhance the image and put it into your OneNote Quick Notes for reference or collaboration. OneNote can be downloaded for free (opens in new tab)<\/span><\/a>.<\/p>\n

OCR accuracy depends on the quality of the image being scanned. Camera-phone users capture images under more diverse conditions than those putting documents through a desktop scanner. Office Lens users might be taking photos from an angle, and photos could be over- or under-exposed, blurred, or suffer from glare from a whiteboard\u2019s reflective surface. This is where the product team\u2019s collaboration with Microsoft Research improved results for Office Lens.<\/p>\n

\u201cWe keep things simple for the user,\u201d says Chris Yu, principal group program manager for Office. \u201cOffice Lens automatically performs image correction and cleanup to photos on your phone before the files are uploaded to OneNote for storage and text conversion. Office Lens detects the edges of the document or whiteboard, or you have the option to set the border manually. You take a photo, and Office Lens cleans up the image and saves it to OneNote, where the OCR software in the cloud does the text-recognition work that allows you to have a searchable digital file.\u201d<\/p>\n

From Whiteboard to OCR<\/h2>\n

The technological starting point for Office Lens came from a research project originally called Whiteboard It (opens in new tab)<\/span><\/a>. Whiteboards provide effective, economical, free-form collaboration, but their contents are difficult to archive and share with colleagues not in the room. Whiteboard image capture and data conversion pose unique challenges. Just ask Zhengyou Zhang<\/a>, principal researcher and research manager of the\u00a0Multimedia, Interaction, and Communication<\/a> group.<\/p>\n

\u201cOur ultimate goal was to reproduce whiteboard content as a faithful, yet enhanced, electronic document,\u201d Zhang says. \u201cThe original project, Whiteboard It, was something I worked on with research engineer Li-wei He (opens in new tab)<\/span><\/a>.\u201d<\/p>\n

\"Office

Office Lens can transform a whiteboard image taken at an angle and exhibiting glare …<\/p><\/div>\n

Whiteboard It identified key challenges in working with whiteboards, such as compensating for perspective distortion when users take photos from an angle, edge detection to localize the board\u2019s boundaries, white balancing to deliver a uniformly white background, and strong color saturation for pen strokes. All these issues and more must be addressed to deliver a crisp image that can be integrated with any Office document.<\/p>\n

Yu learned about Whiteboard It through senior research program manager PD Singh, the liaison between Microsoft Research and Microsoft Office product groups.<\/p>\n

\u201cChris\u2019 team tested the code and very quickly committed to integrating it with Office Lens,\u201d Zhang recalls. \u201cHis group is based in Japan, so we had quite a few online meetings, as well as face-to-face meetings in Redmond, to look into use cases and brainstorm solutions.<\/p>\n

\u201cEven though Whiteboard It is a core technology of Office Lens, it\u2019s just one component of the whole product. For example, the Office Lens product team had to design the best user experience possible for the general public. That\u2019s a lot of work: thinking, designing, and testing. They looked at ways to cope with a person being part of a snapshot of a whiteboard and how to remove that person from the equation. Our technology prototype didn\u2019t include such considerations. I\u2019ve been really impressed with them.\u201d<\/p>\n

Yu, too, is impressed.<\/p>\n

\u201cZhengyou had already envisioned Whiteboard It many years ago,\u201d Yu says. \u201cHe saw the need for an intuitive way to enhance networked meetings. That was back when webcams had just started becoming affordable. Today, device technologies and the cloud have evolved and provide a new context for his work. We are delighted to be delivering his vision.\u201d<\/p>\n

Begin with High-Quality Images<\/h2>\n

Office Lens also incorporates algorithms from Advanced Image Editor (AIE), a project that researcher Lu Yuan<\/a> of Microsoft Research\u2019s Visual Computing Group<\/a> and his colleagues\u2014principal researcher Jian Sun, research development engineer Jiangyu Liu (opens in new tab)<\/span><\/a>, and researcher Kaiming He (opens in new tab)<\/span><\/a>\u2014built to showcase different technologies for improving the quality of consumer photos.<\/p>\n

\"Office

… into a correctly oriented image without any glare.<\/p><\/div>\n

\u201cThe main goal of AIE,\u201d Yuan says, \u201cwas to provide a platform and user interface that made our core technologies easy to present to product teams. In addition, we used AIE to get feedback from internal users and product groups. The feedback they gave us further improved our techniques and drove new ideas. In fact, it was AIE that introduced our advanced image-processing algorithms to the Office team.\u201d<\/p>\n

Within a week of trying out AIE, a product team from Office was back in touch with Yuan to give feedback and to discuss features that would be relevant to their use cases. Yuan and Liu worked closely with product developers to integrate algorithms and code. Then followed an intense second stage of collaboration to test and improve the code, which became part of the Camera Scan (opens in new tab)<\/span><\/a> feature in OneNote for Windows 8.1 (opens in new tab)<\/span><\/a>, released in November 2013. Their work improved existing camera image-capturing functionality for documents and whiteboards via automatic rotation, straightening, cropping, sharpening, and shadow removal.<\/p>\n

\u201cThen we continued our relationship by working with the Office Lens team,\u201d Yuan says. \u201cWe contributed our latest techniques and helped them implement the code.\u201d<\/p>\n

Office Lens automatically applies some of the key image-enhancement algorithms from AIE to correct image tone and exposure, improve contrast, and reduce \u201cnoise.\u201d This might sound like a repeat of Camera Scan, but consider that all of this image-processing functionality had to run efficiently on a mobile device\u2014one of the biggest challenges for Office Lens developers and researchers.<\/p>\n

Perfect Alignment<\/h2>\n

\u201cWorking with a product group is very encouraging,\u201d Yuan says. \u201cIt gives us passion and motivation, and the open dialog helps us understand what\u2019s required for a better user experience. We want to develop techniques in computational photography that help users achieve high-quality photos, so our research goals are in perfect alignment with Office Lens goals.\u201d<\/p>\n

\"Zhengyou_Zhang\"

Zhengyou Zhang<\/p><\/div>\n

Zhang agrees.<\/p>\n

\u201cI am very impressed with the efficacy of Chris\u2019 group,\u201d he says. \u201cOur collaboration with his team was very smooth. They\u2019re open to suggestions and work really hard. I\u2019m thrilled that the technology for Whiteboard It has been productized, and I want to thank the Office Lens team. I truly believe Office Lens will help improve productivity and collaboration.\u201d<\/p>\n

From the product side of the effort, Yu also is thrilled.<\/p>\n

\u201cThe collaboration we\u2019ve had with Microsoft Research for Office Lens has been a natural fit,\u201d he says. \u201cMicrosoft Research has such deep expertise in so many aspects of image enhancement. For example, we were able to consult with Neel Joshi<\/a> and Sebastian Nowozin<\/a> about best practices for blur measurement and Piotr Dollar (opens in new tab)<\/span><\/a> on edge detection.\u201d<\/p>\n

What have early users been saying about Office Lens?<\/p>\n

\u201cFeedback has been incredibly positive,\u201d Yu says. \u201cThey rave about how much time it saves and how it\u2019s changed the way they take notes, now that they know they can just take a photo of a document and find it later in the OneNote cloud. We couldn\u2019t have done this without Microsoft Research.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"

The moment mobile-phone manufacturers added cameras to their devices, they stopped being just mobile phones. Not only have lightweight phone cameras made casual photography easy and spontaneous, they also have changed the way we record our lives. Now, with help from Microsoft Research, the Office team is out to change how we document our lives […]<\/p>\n","protected":false},"author":39507,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[],"msr_hide_image_in_river":0,"footnotes":""},"categories":[194480],"tags":[211853,211856,211838,211832,211835,196748,211841,211844,204607,197784],"research-area":[13551],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-284906","post","type-post","status-publish","format-standard","hentry","category-graphics-and-multimedia","tag-advanced-image-editor-aie","tag-camera-scan-feature","tag-image-correction","tag-office-lens-app","tag-onenote-service-api","tag-optical-character-recognition-ocr","tag-text-recognition","tag-whiteboard-it","tag-windows-phone-8","tag-windows-phone-store","msr-research-area-graphics-and-multimedia","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[247949],"related-projects":[],"related-events":[],"related-researchers":[],"msr_type":"Post","byline":"","formattedDate":"March 17, 2014","formattedExcerpt":"The moment mobile-phone manufacturers added cameras to their devices, they stopped being just mobile phones. Not only have lightweight phone cameras made casual photography easy and spontaneous, they also have changed the way we record our lives. Now, with help from Microsoft Research, the Office…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/284906","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/users\/39507"}],"replies":[{"embeddable":true,"href":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/comments?post=284906"}],"version-history":[{"count":6,"href":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/284906\/revisions"}],"predecessor-version":[{"id":287381,"href":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/284906\/revisions\/287381"}],"wp:attachment":[{"href":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media?parent=284906"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/categories?post=284906"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/tags?post=284906"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=284906"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=284906"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=284906"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=284906"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=284906"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=284906"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=284906"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=284906"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}