@josephj11

the bane of Linux is that you can have almost everything your way, but every time you upgrade or do a clean installl, you have to do it all over again

Oh shit, I don't like that! I had hoped that the configuration files could simply continue to work after an upgrade, or that there was some way to automatically reinstall everything after you make a clean install, with a few preparations of registering your installed modules before the upgrade (ChatGPT suggests dpkg --get-selections > packages.txt, and then later sudo dpkg --set-selections < packages.txt && sudo apt-get update && sudo apt-get dselect-upgrade). Well, I will find out.

Wow, 64 KB RAM, that was huge. 🙂 I started with 16 KB but expanded later to 80 KB, of which the 32 KB had to be switched between two pages by controlling a specific bit. It was fun to tinker with that. And actually fun to be limited, as that sparked creativity. But cool that you started with CP/M; mine was just a toy, but it did teach me all sorts of programming, assembler and even handmade machine code, so I got quite a good grasp on the basics of how computers work. But it sounds fun that you actually built yours from a bare board and parts.

I hope I can get my scanner to work, but yeah, otherwise I guess it is about buying a new one that I know works well with Linux. Thanks for the tip about https://openprinting.github.io/. ... Oh, I just tried, and, perhaps after the command sudo apt-get install cups-backend-bjnp, it worked out of the box, no other install. Now I just need to figure out how to make automatic OCR when scanning to PDF. But I have a sense that there is always a solution to be found.

No, it's not that bad. Configuration files usually carry over in an upgrade unless their format has changed.

A clean install and an upgrade are separate animals.

If you do an upgrade, your optional installed apps carry over too. But so does all the cruft from things you installed just to try and didn't completely remove when you were done. Uninstalling and even purging apps doesn't always get everything, so, sometimes, a clean install is better. It gives you whatever comes with the distro and some things you can optionally add. All with clean, new, uncustomized configuration files.

If you added optional repositories, they get carried over in an upgrade, but they get disabled, so you have to edit them for the new distro version and reenable them.

If you're going with a clean install, then you have to add everything optional that you still want.

I wrote a shell script that takes a list of all the packages that were installed on the old system and installs all of them on the new system. It's smart enough not to install anything that is already installed (which messes up apt a bit), but you have to manually edit the list to eliminate things that shouldn't carry over or that you just don't want any more.

I actually ended up being an assembly programmer as part of my job for seven years. The app was mostly table-driven, but I had some forays into the Z8000 code that made it all work. I loved that job, but it was essentially one long war story. LOL

ChatGPT is almost right. The problem is that a ton of packages are only installed because something else depends on them. These are marked as "automatic". When you uninstall the thing(s) that depended on them, they either also get uninstalled or more usually, become orphans. Apt detects these later and offers you the option to delete them. "Remove unnecessary packages"/"autoremove".

But, when you manually reinstall a package, apt marks it as "manual", so you have to either unmark or remove it yourself and nothing will point that out to you. That's what my script avoids. (available upon request)

I'd have to look up sudo apt-get dselect-upgrade. Never used that before.

Since we're measuring RAM, my first microcomputer was an SD Systems Starter Kit. I added a whole extra bank of RAM to bring it up to 2k. LOL The darn thing had a dual seven segment LED display for output that wasn't even latched. You had to loop to keep the numbers displayed! Everything on that was Z80 machine code.

Learning assembly language took away all the mystique of higher level languages. You knew that if the hardware could do something, you could get it to do it if you could get your code to load and execute.

I have had a running battle with printers and CUPS forever. I even have my own (mostly dead) project - http://sourceforge.net/projects/duplexpr/. I still use it every day, but only for print queue management. I have a duplex printer. I have never done any "automatic" OCR stuff. I have used tesseract, etc. manually.

Back onto upgrades: Theoretically, you should be able to copy your whole install onto a new partition and the do the upgrade there without touching the current "production" version. I worked on that a while quite a few years ago, but couldn't get grub to cooperate. I always partition my main drive with two home and two root partitions and a big data partition so I can do a new clean install while my current version keeps working. I use partition labels in fstab and my scripts so I don't have to deal with UUIDs.

@josephj11 Wow, that was gold. I will use that as a reference going forward.

I have paused the Linux project for a few days while working on my apartment. It's from the 1880s. I'm at the stage of having removed most of the loose and very old paint and loose wall material from a small room, with more or less gaping holes everywhere. That's the stage where everything looks terrible and much worse than when I started. I look forward to when it begins to look orderly again, and hopefully much nicer than when I started. I'm surprised that ChatGPT can even help with that, including identifying what type of walls they are, given pictures of them and the holes as well as the age of the building.

Back to Linux – I am surprised to see how many things I need to get working on Linux, things that I regularly use on Windows but without really being aware of it because the computer is like an extension of me. I have a list of 45 things now, and it's growing. Including such small things like the ability to press Win+Shift+T, select a rectangle on the screen and then have any text in that rectangle put on the clipboard as OCR'ed text – which is surprisingly often useful when text is not selectable for different reasons.

There must be solutions to all these things on Linux, but I have to find a solution to each thing. So I guess the project will take some time, probably months. But I am sure I will find solutions to each thing eventually, and then I will know how to do that. It just occurred to me that it is a bit like moving to another country with a very different culture. Everything is different, but you eventually figure out how to do the different things you want. And a few things you might just have to accept aren't available there.

Learning assembly language took away all the mystique of higher level languages. You knew that if the hardware could do something, you could get it to do it if you could get your code to load and execute.

Yes, it's great to know exactly how it works. At university I even dabbled slightly with microcode, the level below machine code. It's fun to have an understanding of all the levels. I would have loved to program assembly as part of my job, but that didn't happen for me. Professionally it happened to be mostly Python, already from when it was a relatively new language – I fell in love with it and was able to choose to use that.

I was actually thinking that the 'copy text from selected image rectangle' would be relatively easy to make, since it seems like a pipe - (1) take a screenshot (let the user select the rectangle), (2) pipe the result to an OCR machine like Tesseract, (3) put the resulting text on the clipboard. And even if you can't directly pipe it in a command line, you can do something similar, I guess. I just didn't get to it yet. And yes, Autokey can be the thing handling the hotkey to start it. But yeah, making it all in Python would be best, especially if that could potentially make it more cross-platform, but my primary goal is just to get something useful working so I can do the switch to Linux, so it might be a mixture or even just a simple bash script to begin with. But let's see, the devil is in the details. Some sort of easily adjustable settings will be needed, at least for setting the OCR language, I guess, as I tend to deal with several languages (Danish, English, German, French, Swedish, Serbo-Croatian). I don't even know if Tesseract can deal nicely with non-English letters and languages.

Yeah, I was aware of the Python reveal. :-) I just need to get up and running on Linux before I can potentially be productive. Also, my Python has to be dusted off, since I last used it several years ago. But it's still very easy for me to read and understand, as I did get some years under my belt using it. Linux is probably more of a hurdle.

But I do feel very welcome here, so I would like to help if I can. I'm just not fully up and running on Linux yet. I am writing this on Linux, at least, but it's still in a virtual machine.

I have run into a number of situations where having such a script would have been useful. If you create it, I will test it if desired.

I've been on Linux almost forever and would be glad to assist you in getting up to speed wit the CLI and desktop... But, we'll want to do any of that outside of AutoKey. There are a bunch of ways to do that. A few possibilities: I have a private room on Gitter. We could use keet.io, or even email.

You might want to take a look at Warp Terminal. It's a CLI for Linux and Windows with AI built in to assist you.
It will do advanced command completion and even do things when prompted in English. I haven't used it much yet because it won't run on this really old notebook, but it will on my new one that I'm still configuring.

Since you mentioned using multiple (spoken) languages: Words of warning:
AutoKey seems to only work with your keyboard's primary layout. If you switch to another layout, you may be unpleasantly surprised.
If you need to use Unicode characters in AutoKey, there are a bunch of articles in our wiki about how to do that. I'm not sure, but I don't think you can use anything except EN-US characters as abbreviation or hotkey triggers.
Nobody has added language support to AutoKey, so locale changes, etc. will be ignored.

@Elliria I doubt there is any functional difference between the selection and the clipboard. Since everybody uses this functionality across all languages, it should work fine. The problems are that the keyboard module was never upgraded for Unicode support. It's a mystery why this hasn't been addressed upstream by the Python community.

BTW, using subprocess() etc., Python can do anything that bash can do and more, so it just depends on the coder's preference.

I asked Le Chat s:

AutoKey on Linux uses the Python keyboard module to emit keyboard events. This module cannot process Unicode characters correctly. It only works with EN-US characters. Why hasn't the Python community fixed this longstanding issue?


4:31pm

Worked
for
1s
The issue with the Python keyboard module not handling Unicode characters correctly is a known limitation that has persisted for several reasons:

Complexity of Unicode Handling: Unicode support involves dealing with a vast range of characters and different input methods across various operating systems. Implementing comprehensive Unicode support is complex and requires significant effort to ensure compatibility and correctness across different platforms and keyboard layouts[1](https://stackoverflow.com/questions/46271690/detect-keyboard-input-with-support-of-unicode-in-python).

Operating System Dependencies: The keyboard module interacts closely with the operating system's input methods. Different operating systems handle keyboard inputs and Unicode characters differently. For instance, Windows and Linux have different mechanisms for handling keyboard events, and ensuring consistent behavior across these systems adds to the complexity[2](https://pypi.org/project/keyboard/).

Limited Community Contributions: Open-source projects often rely on community contributions to fix issues and add features. The keyboard module may not have received sufficient contributions from developers with the necessary expertise or interest in addressing the Unicode issue. This can result in longstanding issues remaining unresolved[3](https://github.com/moses-palmer/pynput/issues/94).

Priority and Resource Allocation: Maintainers of open-source projects often have to prioritize which issues and features to work on based on available resources and the perceived importance of the task. If Unicode support is not considered a high priority or if the maintainers have limited time to devote to the project, it may not receive the attention needed to resolve the issue[4](https://discuss.python.org/t/installing-keyboard-module-blues/68840).

Alternative Solutions: There may be alternative libraries or methods available that provide better Unicode support, which could reduce the urgency to fix the issue in the keyboard module. Users requiring Unicode support might turn to these alternatives instead of waiting for the keyboard module to be updated[5](https://pypi.org/project/pynput/).

In summary, the lack of comprehensive Unicode support in the keyboard module is due to a combination of technical complexity, limited community contributions, and prioritization challenges. While it remains a known issue, addressing it requires significant effort and resources that have not yet been fully allocated to this task.

Sources

4:31pm



Does the keyboard module in pynput  handle Unicode correctly?



4:37pm
The pynput library is another popular Python library for monitoring and controlling input devices, including the keyboard. However, similar to the keyboard module, pynput also has limitations when it comes to handling Unicode characters.

The pynput.keyboard module can capture keyboard input, but it may not handle Unicode characters correctly, especially on all operating systems. For example, there have been reports of issues when trying to type Unicode characters that do not have a corresponding keysym, such as characters from non-Latin scripts like Bangla[3](https://github.com/moses-palmer/pynput/issues/94).

Here is an example of how you might use pynput to capture keyboard input:

Copy
from pynput import keyboard  
  
def on_press(key):  
    try:  
        print(f'Alphanumeric key pressed: {key.char}')  
    except AttributeError:  
        print(f'Special key pressed: {key}')  
  
def on_release(key):  
    print(f'Key released: {key}')  
    if key == keyboard.Key.esc:  
        # Stop listener  
        return False  
  
# Collect events until released  
with keyboard.Listener(on_press=on_press, on_release=on_release) as listener:  
    listener.join()
In this example, the on_press function will print the character of the key pressed if it is an alphanumeric key. However, for special keys or Unicode characters that do not have a straightforward representation, it may not work as expected.

If handling Unicode input is critical for your application, you might need to explore alternative libraries or methods that are better suited for this purpose. Additionally, you can look into platform-specific solutions that might offer better Unicode support.

Agreed on there probably being no difference between the selection and the clipboard, @josephj11. And I think you're right that the issue is that the keyboard API hasn't been updated to properly work with Unicode.

I don't think it's an upstream issue at all. Python offers a variety of ways to use and find out more about Unicode. A couple of examples are below that specifically show Unicode that's within and outside of the Basic Multilingual Plane (GMP), since many of the emotes we like to use are outside of it:

Example 3-byte Unicode glyph that requires 4 hexadecimal characters because it's within the Basic Multilingual Plane (BMP):

import unicodedata
print(f"""Glyph name: {unicodedata.name("❤")}""")
print(f"""Hex code point: {hex(ord("❤"))}""")
print("\N{Heavy black heart}")
print("\u2764")
print("❤")
print(chr(0x2764))
print(r"\u{}".format("2764").encode().decode('unicode_escape'))
print(u"\u2764".encode('utf-8').decode())

Example 4-byte Unicode glyph that requires 8 hexadecimal characters because it's outside of the Basic Multilingual Plane (BMP):

import unicodedata
print(f"""Glyph name: {unicodedata.name("🐍")}""")
print(f"""Hex code point: {hex(ord("🐍"))}""")
print("\N{snake}")
print("\U0001F40D")
print("🐍")
print(chr(0x1F40D))
print(r"\U{}".format("0001F40D").encode().decode('unicode_escape'))
print(u"\U0001F40D".encode('utf-8').decode())

I don't get this. Yes, Python has ways of dealing with Unicode. Everybody would have stopped using Python years ago if that was the case.

AFAIK, the problem is with the keyboard module, not with all of Python. The AI results I posted above seem to indicate that this is a known, longstanding problem (unless it's a hallucination).

Nice code samples above. I've heard of the astral plane, but this is the first I've heard of the BMP. ;)

I'm pretty new to the BMP, too. It's definitely something in the keyboard API, but as your research showed, handling Unicode is no simple task, so finding developers with experience in all the nuances of dealing with Unicode and enticing them to do the work are the two biggest hurdles.