[-] TheHobbyist@lemmy.zip 1 points 45 minutes ago

Have you installed google services on your phone? they are available through the grapheneOS official "App Store" app. This should be installed before whatsapp is installed (at least that is the instruction for general apps depending on google services).

Perhaps you have done so already, but just a general advice: when using google services and invasive apps like WhatsApp, it can be a good idea to install in their dedicated profile and allow the notifications to pipe through to your main profile instead of installing both in your main profile. If you need help configuring it, let me know.

[-] TheHobbyist@lemmy.zip 1 points 4 days ago

I have no idea if ollama can handle multi-GPU. The 70B in it's q2_k quantized form requires already 26GB of memory, so you would need at least that to run it well and that would only imply it could be entirely run on GPU, which is the best case scenario, but not at what speed.

I know some people with apple silicon who have enough memory to run the 70B model and for them it runs fast enough to be usable. You may be able to find more info about it online.

[-] TheHobbyist@lemmy.zip 1 points 4 days ago

I wish I could. I have an RTX 3060 12GB, I run mostly llama3.1 8B versions in fp8, at 30-35 tokens/s.

[-] TheHobbyist@lemmy.zip 1 points 4 days ago

Sure! It can be a bit of a steep learning curve at times but there are heaps of resources online, and LLMs can also be useful, even if it just in pointing you in the direction for further reading. Regardless, you can reach out to me or other great folks from the !localllama@sh.itjust.works or similar AI, ML or related communities!

Enjoy :)

[-] TheHobbyist@lemmy.zip 3 points 5 days ago* (last edited 5 days ago)

For RAG, there are some tools available in open-webui, which are documented here: https://docs.openwebui.com/tutorials/features/rag They have plans for how to expand and improve it, which they describe here: https://docs.openwebui.com/roadmap#information-retrieval-rag-

For fine-tuning, I think this is (at least for now) out of scope. They focus on inferencing. I think the direction is to eventually help you create/manage your own data which you get from using LLMs using Open-WebUI, but the task of actually fine-tuning is not possible (yet) using either ollama or open-webui.

I have not used the RAG function yet, but besides following the instructions on how to set it up, your experience with RAG may also be somewhat limited depending on which embedding model you use. You may have to go and look for a good model (which is probably both small and efficient to re-scan your documents yet powerful to generate meaningful embeddings). Also, in case you didn't know, the embeddings you generate are specific to an embedding model, so if you change that model you'll have to rescan your whole documents library.

Edit: RAG seems a bit limited by the supported file types. You can get it here: https://github.com/open-webui/open-webui/blob/2fa94956f4e500bf5c42263124c758d8613ee05e/backend/apps/rag/main.py#L328 It seems not to support word documents, or PDFs, so mostly incompatible with documents which have advanced formatting and are WYSIWYG.

[-] TheHobbyist@lemmy.zip 18 points 5 days ago

The interface called open-webui can run in a container, but ollama runs as a service on your system, from my understanding.

The models are local and only answer queries by default. It all happens on the system without any additional tools. Now, if you want to give them internet access, you can, it is an option you have to setup and open-webui makes that possible though I have not tried it myself. I just see it.

I have never heard of any llm "answer base queries offline before contacting their provider for support". It's almost impossible for the LLM to do it by itself without you setting things up for it that way.

[-] TheHobbyist@lemmy.zip 21 points 5 days ago

whats great is that with ollama and webui, you can as easily run it all on one computer locally using the open-webui pip package or in a remote server using the container version of open-webui.

Ive run both and the webui is really well done. It offers a number of advanced options, like the system prompt but also memory features, documents for RAG and even a built in python ide for when you want to execute python functions. You can even enable web browsing for your model.

I'm personally very pleased with open-webui and ollama and they both work wonders together. Hoghly recommend it! And the latest llama3.1 (in 8 and 70B variants) and llama3.2 (in 1 and 3B variants) work very well, even on CPU only, for the latter! Give it a shot, it is so easy to set up :)

[-] TheHobbyist@lemmy.zip 0 points 6 days ago

They don't have to have a backdoor. They are most likely in possession of a master key to decrypt your data:

https://arstechnica.com/gadgets/2012/04/apple-holds-the-master-key-when-it-comes-to-icloud-security-privacy/

31
submitted 1 week ago* (last edited 1 week ago) by TheHobbyist@lemmy.zip to c/linux@lemmy.ml

Hi folks,

I have Alpine Linux installed in an encrypted LUKS partition. I came across this tutorial which shows how to setup a key in a USB drive and when the drive is inserted and the computer booted, the LUKS partition auto-unlocks with the key on the USB drive.

https://askubuntu.com/questions/1414617/configure-ubuntu-22-04-zfs-for-automatic-luks-unlock-on-boot-via-usb-drive

I would like to setup the same thing but I do not have Alpine linux installed on ZFS, so I'm looking for ways to adapt the instructions.

So far, what I've done is:

  1. I've setup the key on the usb stick and I can unlock the LUKS partition with that key.
  2. create a /etc/mkinitfs/features.d/usb-unlock.sh script with the following content:

(the echo to /dev/kmesg was to check whether the script did indeed run at boot by trying to print to the kernel messages but I can't find anything in the kernel messages).

#!/bin/sh

echo "usb-unlock script starting..." > /dev/kmsg

USB_MOUNT="/mnt/my-usb-key" # The USB stick mounting point
LUKS_KEY_FILE="awesome.key"  # The name of your keyfile on the USB stick

# Search for the USB stick with the key
for device in $(ls /dev/disk/by-uuid/*); do
    mount $device $USB_MOUNT 2>/dev/null
    if [ -f "$USB_MOUNT/$LUKS_KEY_FILE" ]; then
        # Unlock the LUKS partition
        cryptsetup luksOpen /dev/sda3 cryptroot \
            --key-file "$USB_MOUNT/$LUKS_KEY_FILE" && exit 0
    fi
    umount $USB_MOUNT
done
echo "No USB key found, falling back to password prompt." # this message never appears, despite not having found the key on the usb stick

echo "usb-unlock script ending." > /dev/kmsg
  1. I added usb-unlock to the features in mkinitfs.conf:
mytestalpine:~# cat /etc/mkinitfs/mkinitfs.conf 
features="ata base ide scsi usb virtio ext4 cryptsetup keymap usb-unlock"
  1. run mkinitfs to rebuild the initramfs. Then reboot to test the implementation, which was unsuccessful.

What am I missing / doing wrong? Thank you for your help!

Edit: forgot to add step 4

14
submitted 2 months ago by TheHobbyist@lemmy.zip to c/privacy@lemmy.ml

Hi folks,

I'm seeing there are multiple services which externalise the task of "identity provider" (e.g. login with Facebook, google or what not).

In my case, I am curious about Tailscale, a VPN service which allows one to chose an identity provider/SSO between Google, Microsoft, Github, Apple and OIDC.

How can I find out what data is actually communicates to the identity provider? Their task should simply be to decide whether I am who I claim to be, nothing more. But I'm guessing there may be some subtleties.

In the case of Tailscale, would the identity provider know where I'm trying to connect? Or more?

Answers and insights much appreciated! The topic does not seem to have much information online.

6
submitted 2 months ago* (last edited 2 months ago) by TheHobbyist@lemmy.zip to c/selfhost@lemmy.ml

Hi folks, I'm considering setting up an offsite backup server and am seeking recommendations for a smallish form factor PC. Mainly, are there some suitable popular second hand PCs which meet the following requirements:

  • fits 4x 3.5" HDD
  • Smaller than a regular tower (e.g. mATX or ITX)
  • Equipped with a 6th of 7th gen Intel CPU at least (for power efficiency and transcoding, in case I want it to actually to some transcoding) with video output.
  • Ideally with upgradeable RAM

Do you know of something which meets those specs and is rather common on the second hand market?

Thanks!

Edit: I'm looking for a prebuilt system, such as a dell optiplex or similar.

43
submitted 3 months ago* (last edited 3 months ago) by TheHobbyist@lemmy.zip to c/foss@beehaw.org

Yesterday, there was a live scheduled by Louis Grossman, titled "Addressing futo license drama! Let's see if I get fired...". I was unable to watch it live, but now the stream seems to be gone from YouTube.

Did it air and was later removed? Or did it never happen in the first place?

Here's the link to where it was meant to happen: https://www.youtube.com/watch?v=HTBYMobWQzk

Cheers

Edit: a new video was recently posted at the following link: https://www.youtube.com/watch?v=lCjy2CHP7zU

I do not know if this was the supposedly edited and reuploaded video or if this is unrelated.

28
submitted 6 months ago* (last edited 6 months ago) by TheHobbyist@lemmy.zip to c/localllama@sh.itjust.works

From Simon Willison: "Mistral tweet a link to a 281GB magnet BitTorrent of Mixtral 8x22B—their latest openly licensed model release, significantly larger than their previous best open model Mixtral 8x7B. I’ve not seen anyone get this running yet but it’s likely to perform extremely well, given how good the original Mixtral was."

[-] TheHobbyist@lemmy.zip 92 points 6 months ago

You can put up a non commercial license and write that if this is for a commercial application they can get in touch with you and you can discuss together a new license for their use case.

13
submitted 10 months ago by TheHobbyist@lemmy.zip to c/steamdeck@sopuli.xyz

I was exploring the fps and refresh rate slider and I realized that when setting the framerate limiter to 25, the refresh rate was incorrectly set to 50Hz on the OLED version, when the 75 Hz setting would be a more appropriate setting, for the same reason 30 fps is at 90 Hz and not 60 Hz. Anyone else seeing the same behavior? Is there an explanation I'm missing here?

8

Hi folks, I'm looking for a specific YouTube video which I watched around 5 months ago.

The gist of the video is that it was comparing the transcoding performance of an Intel iGPU when used natively, compared to when passed through to a VM. From what I recall there was a significant performance hit and it was around 50% or so (in terms of fps transcoding). I believe the test was performed on jellyfin. I don't remember whether it was using xcpng, proxmox or another OS. I don't remember which channel published this video nor when it was published, just that I watched it sometime between April and June this year.

Anyone recall or know what video I'm talking about? Possible keywords include: quicksync, passthrough, sriov, iommu, transcoding, iGPU, encoding.

Thank you in advance!

[-] TheHobbyist@lemmy.zip 91 points 1 year ago

I hear you but this seems to largely ignore that we are all already paying google, a lot. It is only thanks to their unscrupulous private data harvesting that they have become the mastodon they are. This has been going on for so long and only in the recent past to we get the scale of this effort. Now they want us to pay them too, while nothing is changing on the data privacy side? Frankly, I don't think they deserve our trust. It's not like paying makes them get any less of our private data, so they are basically double dipping. That does not sit well with me.

I'm all for paying for a due service, but I also have expectations of data privacy rights. Those are mostly vanishing into thin air with google...

18
submitted 1 year ago* (last edited 1 year ago) by TheHobbyist@lemmy.zip to c/selfhosted@lemmy.world

Hi y'all,

I am exploring TrueNAS and configuring some ZFS datasets. As ZFS provides with some parameters to fine-tune its setup to the type of data, I was thinking it would be good to take advantage of it. So I'm here with the simple task of choosing the appropriate "record size".

Initially I thought, well this is simple, the dataset is meant to store videos, movies, tv shows for a jellyfin docker container, so in general large files and a record size of 1M sounds like a good idea (as suggested in Jim Salter's cheatsheet).

Out of curiosity, I ran Wendell's magic command from level1 tech to get a sense for the file size distribution:

find . -type f -print0 | xargs -0 ls -l | awk '{ n=int(log($5)/log(2)); if (n<10) { n=10; } size[n]++ } END { for (i in size) printf("%d %d\n", 2^i, size[i]) }' | sort -n | awk 'function human(x) { x[1]/=1024; if (x[1]>=1024) { x[2]++; human(x) } } { a[1]=$1; a[2]=0; human(a); printf("%3d%s: %6d\n", a[1],substr("kMGTEPYZ",a[2]+1,1),$2) }'

Turns out, that's when I discovered it was not as simple. The directory is obviously filled with videos, but also tiny small files, for subtitiles, NFOs, and small illustration images, valuable for Jellyfin's media organization.

That's where I'm at. The way I see it, there are several options:

    1. Let's not overcomplicate it, just run with the default 64K ZFS dataset recordsize and roll with it. It won't be such a big deal.
    1. Let's try to be clever about it, make 2 datasets, one with a recordsize of 4K for the small files and one with a recordsize of 1M for the videos, then select one as the "main" dataset and use symbolic links for each file to the other dataset such that all content is "visible" from within one file structure. I haven't dug too much in how I would automate it, but might not play nicely with the *arr suite? Perhaps overly complicated...
    1. Make all video files MKV files, embed the subtitles, rename the videos to make NFOs as unnecessary as possible for movies and tv shows (though this will still be useful for private videos, or YT downloads etc)
    1. Other?

So what do you think? And also, how have your personally set it up? Would love to get some feedback, especially if you are also using ZFS and have a videos library with a dedicated dataset. Thanks!

Edit: Alright, so I found the following post by Jim Salter which goes through more detail regarding record size. It clarifies my misconception about recordsize not being the same as the block size, but also it can easily be changed at any time. It's just the size of the chunks of data to be read. So I'll be sticking to 1M recordsize and leave it at that despite having multiple smaller files, because the important will be to effectively stream the larger files. Thank you all!

view more: next ›

TheHobbyist

joined 1 year ago