Improving Joplin's Image Rendering

Published , updated .
Keywords: software-development joplin

Background

When pasting screenshots from Mac OS, they are shown at 2x resolution in Joplin (and in rendered GitHub pages) since that’s how Mac OS captures them on a retina screen.

If you take a screenshot on Mac OS and open Preview, then create a new document with the contents of the clipboard, the resulting .png file will be twice the width and twice the height of the original screenshot.

Critically though, when pasting into Preview, Mac OS recognises that the Image DPI is 144 pixels/inch (instead of the regular 72 pixels/inch).

Here’s a 70 x 70 screenshot I took:

Pasting into Preview the Info panel shows a 140 x 140 pixel image, but recognizes the image DPI.

Opening the image after it has been pasted into Joplin (by right clicking on the image and selecting Reveal file in Finder) shows that the image DPI information has been lost.

TL ; DR

It seems like Electron, on which Joplin is based, does not preserve image EXIF data when reading from the Clipboard.

There is some ongoing work to improve Electron’s clipboard API which may address this issue: https://github.com/electron/governance/pull/454

The above pull request references the following issue, created in April 2020 by fabiospampinato (author of Notable) that requests better support for reading images from the clipboard. https://github.com/electron/electron/issues/23156

Potential Solutions

Some thoughts on how to address this issue.

Find a screenshot app that rescales screenshots

It’s likely that an app exists on Mac OS that will take and rescale screenshots automatically.

See whether Joplin can identify high-DPI screenshots

Perhaps Joplin could intercept the paste command and rescale the image before saving its copy?

See whether EXIF data would allow the Joplin renderer to add width and height attributes

Perhaps there is something in the EXIF data of the png file that would allow Jopiln’s img renderer to identify that width and height attributes should be added at half the images natural resolution?

To figure out

Is there anything in the EXIF (or other) data from a screenshot that tells you it has been taken at double-DPI?

Summary: YES, a DPI of 144 is stored in the image’s EXIF data.

After taking a screenshot, saving as a PNG, it simply shows up as having double sized dimensions in the Mac OS file info screen.

Using the file utility doesn’t show the DPI:

file screenshot-1.png
screenshot-1.png: PNG image data, 232 x 250, 8-bit/color RGBA, non-interlaced

But in the general tab of Preview, you can see the DPI is 144:

ImageMagick’s identify tool shows that the DPI is stored in the image’s EXIF data:

identify -format '%[EXIF:*]' screenshot-1.png
exif:ExifOffset=90
exif:Orientation=1
exif:PixelXDimension=232
exif:PixelYDimension=250
exif:ResolutionUnit=2
exif:UserComment=65, 83, 67, 73, 73, 0, 0, 0, 83, 99, 114, 101, 101, 110, 115, 104, 111, 116
exif:XResolution=144/1
exif:YResolution=144/1

But importantly, after pasting into Joplin, the EXIF data is lost:

identify -format '%[EXIF:*]' c41d8c9e49774020a93ac6956112b0b6.png

So even if the web browser could identify the DPI (can it?), it won’t be able to since it’s not preserved.

What does the Mac OS clipboard API look like? When the clipboard stores a screenshot, does it show pixel density?

Research

How can I stop my retina display from taking 2x sized screenshots? https://apple.stackexchange.com/questions/105185/how-can-i-stop-my-retina-display-from-taking-2x-sized-screenshots

EXIF https://en.wikipedia.org/wiki/Exif

How do I take a MacOS Retina screenshot and get the image at its actual size? https://o7planning.org/12711/how-do-i-take-a-macos-retina-screenshot-and-get-the-image-at-its-actual-size

  • A tutorial showing how to use Automator to save resized versions of screenshots.

Searching for EXIF on Joplin’s Subreddit, no results: https://www.reddit.com/r/joplinapp/search/?q=exif&restrict_sr=1&sr_nsfw=

Searching for EXIT on Joplin’s support forum: https://discourse.joplinapp.org/search?q=exif

This application, Maccy, remembers clipboard history on Mac OS. Perhaps we can look at the source code to see what the Mac OS clipboard API looks like? https://github.com/p0deje/Maccy

Similar Joplin GitHub issues

Images sometimes improperly rotated #3383 https://github.com/laurent22/joplin/issues/3383

  • This issue also related to EXIF data.

Keep original image rather than compressing, please #1232 https://github.com/laurent22/joplin/issues/1232

  • This issue also has a discussion of EXIF data.
  • It points to the following part of the code which may be relevant: ./ReactNativeClient/lib/shim-init-node.js:141-148:

Examining Joplin source code

Building the Mac OS Desktop client by following https://github.com/laurent22/joplin/blob/dev/BUILD.md.

After taking a little while on yarn install and finishing first time with errors and second time with warnings, loading the desktop app works:

Electron’s NativeImage Class

Some of the code to handle image resizing is here:

packages/lib/shim-init-node.js line 162

let image = nativeImage.createFromPath(filePath);

It uses Electronic’s NativeImage class: https://www.electronjs.org/docs/latest/api/native-image/#imageisempty

Clipboard data is retrieved using Electron’s clipboard.readImage: https://www.electronjs.org/docs/latest/api/clipboard/#clipboardreadimagetype

Electron Issues

OPEN: Clipboard: improved support for reading images #23156 https://github.com/electron/electron/issues/23156

  • The issue author is writing a note taking app.
  • The issue is around preserving original clipboard data.