Today I’m going to talk about Yarn.social’s future, a roadmap into where we’re going and thinking. I’ll also write a little about it’s history of where we came from and highlight how Yarn.social is different and in my opinion better.

What?

For those of you new to my blog or Yarn.social; Yarn.social is a decentralised social media platform, a microBlogging platform. It was originally created in July of 2020 and was then called Twt.social – since then it has going through a rebranding and seen huge improvements in the user experience and features that make up the platform.

If you are somewhat familiar or used to Twtter™, Yarn.social has a similar feel, but the key things to note are:

• It is 100% decentralised, there is no single server, entity or organisation in “control”
• There is no metadata collection of any kind or tracking of users
• There are absolutely no advertising

Where?

Today Yarn.social consists of:

• An ever improving backend called yarnd that powers every single and multi user pod (what we call instances that make up the network)
• An ever improving default theme providing a decent user experience for all new pod owner/operators and users
• A new but basic landing page at Yarn.social that has most of the content we need in place
• An updated native Mobile App actively being worked on but that is still in internal development and alpha testing
• A global search engine and crawler that whilst functional still needs some work
• A feed service that supports subscribing to Twitter and RSS/Atom feeds
• 4x multi-user pods and 7 single-user pods
• 415 feeds across the network and general ecosystem which have produced and consumed some whopping ~256,000 Yarns!

When?

Tomorrow it is my hope that we:

• Continue to improve the general user experience of the default web interface’s theme
• Complete the re-development of the native Mobile App and get it published to the App and Play stores
• Redesign the landing page at Yarn.social with a better layout and look

Not literally tomorrow of course 😂 but this is more of an ongoing set of efforts I’d like to see continue as well as some immediate long-over due goals.

In addition to the above, I would really like to get a Not-for-Profit (NFP) organisation setup and created for Yarn.social – Why? I believe Yarn.social and all the work we’ve put into the platform, it’s components, it’s specs and so forth should remain wholly in the hands of the community and in my opinion we should continue to govern ourselves this way. I never intended to create yet another privacy eroding social platform (you know the ones that profit off of user data and metadata and care nothing but user engagement on their centralised platform so drive up advertising revenue).

So over the holidays:

• Seek help in setting up a Not-for-Profit

Or:

• Sit down and work through the required paperwork and just “get it done”

Once this is done we can move on to other things that are in the works and eventual long-term goals such as:

• Getting Yarn.social yarnd into the Marketplace of Cloud providers such as Vultr and DigitalOcean – The only blocker right now is an official organisation and support structure (hence the need and desire to create a Not-for-Profit above).
• Seeking funding from either the community through donations and/or through philanthropy investors so that we can continue to support the development and maintenance of the platform over time.
• Redesigning and developing the Hosted Pods platform as a service to create a level of convenience for new users and new pod owner/operators that either cannot run/host their own pod due to lack of experience or technical adeptness or prefer to have someone else manage the “hosting” and “maintenance”.
• Building a new and more modern sleeker Web Application frontend ala Race-style (but without using any heavy frameworks like React!) – This is a bit controversial (or has been) for some, especially those that enjoy the lightweight ness of the server-side web interface that exists on all pods today. But it becomes difficult to build some types of user experiences this way and it is also hard to maintain a native Mobile App experience that uses a completely different set of technologies, codebase and even programming language! The idea here (and has been already started somewhat) is to build a Progressive Web App / Single Page App that utilises the backend API of yarnd and Mithril JS as well as some good CSS to create a rich, interactive, almost-native app-like experience. This app would be hosted centrally at either app.yarn.social or yarn.social/app and serve as a front to any pod, it may also be builtin to yarnd itself as an optional web interface.

And finally, but most importantly to me:

• Continue the discussion of building and supporting the idea of “Private Feeds” – We’ve talked about the use-cases for this over some months, and whilst this itself may be a lot of work, and there may be other ways to solve similar use-cases, I feel it is important to also build ways in which people are also able to “socialise” and “microBlob” privately.
• Reignite a much older discussion and idea of (what was originally called) “Supervised Pods”, but more recently I’ve thought of this as “Supervised Accounts”. This is the idea that (and something very important to me) that as a Father of two young daughters who will very likely want to get into such things as “social media”, that they should be able to do so BUT not on any of these other privacy eroding garbage that exists out there AND should be protected from predators, bullying, harassment and so forth (things that are in violation of all pod’s default abuse policy anyway and would result in a Pod Admin/Operator just nuking your account).

And so this concludes the general vision and goals of Yarn.social. Over time it is my great hope that the network continues to grow and continues to be cultivated. IT’s hard to put numbers on this because of the decentralised nature of Yarn.social / Twtxt in the first place; but let’s see if we can (over the next year) get to:

• 20+ Yarn.social pods
• 500+ Feeds/Users across the network and ecosystem
• 1M+ produced and consumed Yarns!

How?

Good question! This is a lot of work, it’s already been a lot of work. In only a few short months Yarn.social itself will reach it’s 1st official birthday (2nd if you count the fact it was once called Twt.social).

This can go either one of two ways in my view:

• Yarn.social continues to evolve and improve slowly over time with the help of contributors that donate what little spare time they have (including myself)

Amd/Or:

• I start to wind down my own day job over time so focus more on the things I love. I obviously can’t do this yet as I still need to also support my family.

And/Or:

• A nice philanthropist comes along and donates a sizeable chunk of money to the project because they see the value in what we’re building and then suddenly things go a lot faster! 😂

Why?

Many of the driving factors behind Yarn.social today that were originally talked about in a much older post Why an open, decentralised social media network like Twtxt and Twt.social matter are still true today.

I won’t restate in great detail what is already on the About Yarn.social page, but if I am to sum up everything into a single visionary statement it would be:

To create the best, unobtrusive and unfettered social medial platform to anyone that values their privacy and freedoms.

This is a blog post on why I no longer trust Github that Microsoft bought a few years ago (for a few billion) with my code. I will outlines the events that occurred, the thinking that lead me to the decisions that I made and my final thoughts.

Rest assured, I will continue to contribute to open source, write open source software, I just won’t host it on any corporate owned “Cloud” (SaaS) code hosting platform. I will self-host*.

A little about me

Those of you who don' know, my name is James Mills. I’ve been with Github since ~2008. I started contributing to open source and writing open source projects since ~2004 with my first project being a Python library that later turned into the circuits event-driven asynchronous library used by many (well before asyncio became a thing in Python many years later).

I’ve worked in numerous companies in Australia being a Backend Developer, Full Stack Engineer, DevOps Engineer, System Administrator, Network Administrator, Helpdesk Support and many other roles over the years. I’ve worked at Facebook for over 3 years as a Production Engineer where I earned much respect for my ability to “Get Stuff Done"™.

Over the course of some ~17 years of contributing to and writing open source projects I roughly estimate (really hard to actually measure) I’ve written between 500,000 to 1,000,000 lines of code.

Why did I leave Github?

I am primarily leaving Github because I no longer trust them as an entity to host my code. Code that I write and openly license under the MIT License. The recent announcement of Microsoft/Github’s Copilot which harvested source code from public Github repositories and used that as the training dataset for their machine learning models I feel is both a violation of the (©) Copyright I hold on source code I wrote but also a violation of the License agreement(s) I typically choose to license my works under.

Specifically the use of any of my works is subject to (because I use the same license normally):

Permissions

• Commercial use
• Modification
• Distribution
• Private use

Limitations

• Liability
• Warranty

Conditions

• License and copyright notice

It is the last one that is really important to me. You are free to use my works in any projects of your own, commercial or otherwise, as long as you attribute my work and maintain my License and Copyright. If you do not you are violation.

As a user of Copilot, if any works you derive (or in some cases is a direct copy/pasta) from my source code is in direction violation of the license of my work unless you:

• Know the source of the suggestions and snippets that came from Copilot
• AND acknowledge, attribute and maintain the License and Copyright of the original works.

Github’s own Terms of Service even states (if you bother to read it carefully):

This license does not grant GitHub the right to sell Your Content. It also does not grant GitHub the right to otherwise distribute or use Your Content outside of our provision of the Service, except that as part of the right to archive Your Content, GitHub may permit our partners to store and archive Your Content in public repositories in connection with the GitHub Arctic Code Vault and GitHub Archive Program.

Assuming Microsoft who wrote Copilot, a Visual Studio extension, intends to eventually sell this as a service to their users, they are in direct violation of Github’s own Terms of Service as stated above and captured here for posterity.

FWIW I used to up until now even pay Github for “extra features” and “privileges” on their platform. Things like unlimited private repositories, more actions, more storage, etc. No longer! I’ve now downgraded to the “Free” account.

So what happened?

Here follows a series of events, including screenshots for posterity (in case things get removed or re-worded, etc) of what happened over the last couple of weeks that eventually lead me to the decision to remove all of my source code from Github.

• Sun, Jul 4 3:34AM Slashdot posts about mixed reactions of Micorosft’s Copilot

• Sun, Jul 4 5:21AM A fellow open source developer talks about “Abandoning GitHub”

• Sun 4, 12:09PM I followup (_after reading about what’s ) with thoughts of my own and contemplating follow suit.

• Mon 5, 9:10PM Julia Reda posts about GitHub Copilot is not infringing your copyright.

  I don’t know who this Julia Reda is, but even though she’s right about Github/Microsoft not technically infringing on (©) Copyright per se, what has been (and is being done) is a direction violation of many open source licenses.

• Wed 7, 5:07PM I ask my social media and those that follow me their opinions.

• Thu 8, 6:18PM Github confirm that they didn’t just selectively use a subset of repositories with particular licenses, they scraped them all!

  

• Fri 9, 11:57PM Another open source developer announces leaving Github over the Copilot launch.

• 3:30PM I finally announce my decision to do the same.

So now what?

So now that I’ve migrated all of my projects and their source code off of Github to my own Self-Hosted platform, then deleted all content from Github (except my identity, which I will maintain); Now what?

Projects have moved

All my projects have a new home at git.mills.io. I’ve always had my own private Git Hosting platform (I used to use Gogs) but since this weekend I’ve decided to stand up Gitea instead as it has a few more features that make it a bit more convenient for collaborating with others and has a familiar feel and experience as compared with Github/Gitlab/etc.

I’ve also enabled Github OAuth as well so anyone that had previously contributed to my projects on Github, raised Issues or just want to continue to follow me work can do so with relative ease by simply signing in to https://git.mills.io with their Github identity.

Go libraries and import paths

Since I’ve been an avid Go developer for quite some years now and have written a considerable number of libraries and software (some quite popular), I feel you should be aware that import paths are now obviously broken and you need to update your own projects:

Change any import from:

import "github.com/prologic/<library>"

To:

import "git.mills.io/prologic/<library>"

At some point I will be standing up my own Go Vanity Package hosting to make the imports canonical, simpler and resistant to change no matter where the code lives.

Final thoughts

It’s been a crazy couple of years with the recent pandemic and all. Something fairly similar happened to a chat platform I had been a member of for over ~18 years, FreeNode IRC Network. It never ceases to amaze me what corporations can do to ruin a “good thing”.

I want to thank each and every one of you that have contribute to my own projects and accepted my own contributions via Github over the years. I hope that many of you will continue to follow my work and use many of my works as I continue to maintain and develop it under it’s new home (mine!).

Happy hacking! 🙇‍♂️

II may also consider deleting all my source code repositories and forks from Github.

For more details see: twtxt.net/conv/xv2o…

Welcome to comment here also, I will respond here too!

So I’m a Knucklehead eh?

This small post is mostly in response to an article I saw this morning in my feeds: Building a search engine to rival Google could cost billions — and that’s not the only problem by James Purtill.

Challenge accepted!

I challenge anyone with a few spare tens of millions of $AUD to come talk to me about this. Put your money where your mouth is (so to speak). Does it really cost 100’s of Billions of dollars ro build, crawl and index the Web? I think not!

What, wait? Who are you?!

Who the hell am I you ask? Really?! You’re crazy!

Maybe 😉

I don’t expect most of you to know me too well (except those that do!), I’m just a software engineer that has worked in quite a few places over a period of more than 15 years, including several years at Facebook Inc. I’m very familiar with the kinds of Infrastructure and Engineering it takes to build “Web Scale” products like Search or a large-centralised Social Media!

It is quite sad to read other failed attempts at building a search engine, mostly Government backed attempts. The reality is a lot of government backed projects tend to be inefficient and the funds wasted before the project even has a chance to get off the ground or suffers from political issues (example: NBN Co in Australia where many of us still have Copper Last Mile over *DSL).

What would it take?

What would it cost to build an Aussie search engine? Well I put together one over a weekend called Spyda because my wife asked me to! I said I’d have it ready by Monday. It was a day late and launched on Tuesday.

Spyda is available at spyda.dev where you can currently search a measly 1,000+ pages. We’ve only been able to crawl and inex a thousand pages so far. This is becuase it runs on my home infrastructure which has an ordinary 100/40 Mbps VDSL NBN Co connection backing it (behind Cloudflare).

In other words it cost a day of my engineering time to put together, plus the cost of my home infrastructure (with initial outlay of $10k and $5k maintenance over the years).

So basically nothing so far to prove the concept is feasible. 😝

What now?

So let’s scale this up!

If someone were to approach me tomorrow offering to invest in my expertise through a new company founded to compete with the likes of Google, Facebook, Twitter and all these other tech-giants that have successfully built up a multi-Trillion dollar empire by profiting off the data collected from their “users”, come talk to me!

Let’s build an Aussie search engine, that’s Aussie built and owned, but does not sell its users data!

You can reach me at James dot Mills at shortcircuit dot net dot au – Or just Google™ me 🤣

Learn Go in ~5mins

This is inspired by A half-hour to learn Rust and Zig in 30 minutes.

Basics

Your first Go program as a classical “Hello World” is pretty simple:

First we create a workspace for our project:

$ mkdir hello

Next we create and initialize a Go module:

$ go mod init hello

Then we write some code using our favorite editor:

package main

import "fmt"

func main() {
  fmt.Println("Hello World!")
}

And finally we build and produce a binary:

$ go build

You should now have a hello binary in your workspace, if you run it you should also get the output:

$ ./hello
Hello World!

Variables

You can create variables in Go in one of two ways:

var x int

Other types include int, int32, int64, float32, float64, bool and string (and a few others…), there are also unsigned variants of the integer types prefixed with u, e.g: uint8 which is the same as a byte.

Or implicitly with inferred types by creating and assigning a value with:

x := 42

Values are assigned by using the = operator:

x = 1

Functions

Functions are declared with the func keyword:

func hello(name string) string {
  return fmt.Sprintf("Hello %s", name)
}

Functions with a return type must explicitly return a value.

Functions can return more than one value (commonly used to return errors and values):

func isEven(n int) (bool, error) {
  if n <= 0 {
    return false, fmt.Errorf("error: n must be > 0")
  }
  return n % 2 == 0, nil
}

Go also supports functions as first-class citizens and as such supports many aspects of functional programming, including closures, returning functions and passing functions around as values. For example:

func AddN(n int) func(x int) int {
	return func(x int) int {
		return x + n
	}
}

Structs

As Go is a multi-paradigm language, it also support “object orientated” programming by way of “structs” (borrowed from C). Objects / Structs are defined with the struct keyword:

type Account struct {
  Id:      int
  Balance: float64
}

Fields are defined similar to variables but with a colon : separating their name and type. Fields are accessed with the dot-operator .:

account := Account{}
fmt.Println("Balance: $%0.2f", account.Balance)

Methods

Structs (objects) can also have methods. Unlike other languages however Go does not support multiple-inheritance nor does it have classes (you can however embed structs into other structs).

Methods are created like functions but take a “receiver” as the first argument:

type Account struct {
  id  int
  bal float64
}

func (a *Account) String() string {
  return fmt.Sprintf("Account[%d]: $0.2f", a.id, a.bal)
}

func (a *Account) Dsposit(amt flaot64) float64 {
  a.bal += amt
  return a.bal
}

func (a *Account) Withdraw(amt float64) float64 {
  a.bal -= amt
  return a.bal
}

func (a *Account) Balance() float64 {
  return a.bal
}

These are called “pointer receiver” methods because the first argument is a pointer to a struct of type Account denoted by a *Account.

You can also define methods on a struct like this:

type Circle struct {
  Radius float64
}

func (c Circle) Area() float64 {
  return 3.14 * c.Radius * c.Radius
}

In this case methods cannot modify any part of the struct Circle, they can only read it’s fields. They are effectively “immutable”.

Arrays and Slices

Arrays are created with [T]like this:

var xs []int = []int{1, 2, 3, 4}

Arrays can also be created and appended to:

var xs []int
xs = append(xs, 1)
xs = append(xs, 2)
xs = append(xs, 3)

You can access an array’s elements by indexing:

xs[1]  // 2

You can also access a subset of an array by “slicing” it:

ys := xs[1:] // [2, 3]

You can iterate over an array/slice by using the range keyword:

for i, x := range xs {
  fmt.Printf("xs[%d] = %d\n", i, x)
}

Maps

Go has a builtin data structure for storing key/value pairs called maps (called hash table, hash map, dictionary or associative array in other languages).

You create a map by using the keyword map and defining a type for keys and type for values map[Tk]Tv, for example a map with keys as strings and values as integers can be defined as:

var counts map[string]int

You can assign values to a map just like arrays by using curly braces {...} where keys and values are separated by a colon :, for example:

var counts = map[string]int{
  "Apples": 4,
  "Oranges": 7,
}

Maps can be indexed by their keys just like arrays/slices:

counts["Apples"]  // 4

And iterated over similar to array/slices:

for key, value := range counts {
  fmt.Printf("%s: %d\n", key, value)
}

The only important thing to note about maps in Go is you must initialize a map before using it, a nil map will cause a program error and panic:

var counts map[string]int
counts["Apples"] = 7  // This will cause an error and panic!

You must initialize a map before use by using the make() function:

var counts map[string]int
counts = make(map[string]int)
counts["Apples"] = 7

Flow control structures

Go only has one looping construct as seen in the previous sections:

sum := 0
for i := 0; i < 10; i++ {
  sum += i
}

The basic for loop has three components separated by semicolons:

• the init statement: executed before the first iteration
• the condition expression: evaluated before every iteration
• the post statement: executed at the end of every iteration

If you omit the condition you effectively have an infinite loop:

for {
}
// This line is never reached!

Go has the usual if statement along with else if and else for branching:

var N = 42
func Guess(n int) string {
  if n == 42 {
    return "You got it!"
  } else if n < N {
    return "Too low! Try again..."
  } else {
    return "Too high! Try again..."
  }
}        

Note: The last else could have been omitted and been written as return "Too high~ Try again...", as it would have been functionally equivalent.

There is also a switch statement that can be used in place of multiple if and else if statements, for example:

func FizzBuzz(n int) string {
  switch n {
  case n % 15 == 0:
    return "FizzBuzz"
  case n % 3 == 0:
    return "Fizz"
  case n % 5 == 0:
    return "Buzz"
  default:
    return fmt.Sprintf("%d", n)
  }
}

Functions can be executed at the end of a function anywhere in your function by “deferring” their execution by using the defer keyword. This is commonly used to close resources automatically at the end of a function, for example:

package main

import (
  "os"
  "fmt"
)

func Goodbye(name string) {
  fmt.Printf("Goodbye %s", name)
}

func Hello(name string) {
  defer Goodbye(name)
  fmt.Printf("Hello %s", name)
}

func main() {
  user := os.Getenv("User")
  Hello(user)
}

This will output when run:

$ ./hello
Hello prologic
Goodbye prologic

Error handling

Errors are values in Go and you return them from functions. For example opening a file with os.Open returns a pointer to the open file and nil error on success, otherwise a nil pointer and the error that occurred:

f, err := os.Open("/path/to/file")

You check for errors like any other value:

f, err := os.Open("/path/to/file")
if err == nil {
  // do something with f
}

It is idiomatic Go to check for non-nil errors from functions and return early, for example:

func AppendFile(fn, text string) error {
  f, err := os.OpenFile(fn, os.O_CREATE|os.O_APPEND|os.WR_ONLY, 0644)
  if err != nil {
    return fmt.Errorf("error opening file for writing: %w", err)
  }
  defer f.Close()
  
  if _, err := f.Write([]byte(text)); err != nil {
    return fmt.Errorf("error writing text to fiel: %w", err)
  }
  
  return nil
}

Creating and import packages

Finally Go (like every other decent languages) has a module system where you can create packages and import them. We saw earlier In Basics how we create a module with go mod init when starting a new project.

Go packages are just a directory containing Go source code. The only difference is the top-line of each module (each *.go source file):

Create a Go package by first creating a directory for it:

$ mkdir shapes

And initializing it with go mod init:

$ cd shapes
$ go mod init github.com/prologic/shapes

Now let’s create a source module called circle.go using our favorite editor:

package shapes

type Circle struct {
  Radius float64
}

func (c Circle) String() string {
  return fmt.Sprintf("Circle(%0.2f)", c.Radius)
}

func (c Circle) Area() float64 {
  return 3.14 * c.Radius * c.Radius
}

It is important to note that in order to “export” functions, structs or package scoped variables or constants, they must be capitalized or the Go compiler will not export those symbols and you will not be able access them from importing the package.

Now create a Git repository on Github called “shapes” and push your package to it:

$ git init
$ git commit -a -m "Initial Commit"
$ git remote add origin git@github.com:prologic/shapes.git
$ git push -u origin master

You can import the new package shapes by using it’s fully qualified “importpath” as github.com/prologic/shapes. Go automatically knows hot to fetch and build the package given its import path.

Example:

Let’s create a simple program using the package github.com/prologic/shapes:

$ mkdir hello
$ go mod init hello

And let’s write the code for main.go using our favorite editor:

package main

import (
	"fmt"

	"github.com/prologic/shapes"
)

func main() {
	c := shapes.Circle{Radius: 5}
	fmt.Printf("Area of %s: %0.2f\n", c, c.Area())
}

Building it with go build:

$ go build

And finally let’s test it out by running the resulting binary:

$ ./hello
Area of Circle(5.00): 78.50

Congratulations! 🎉

Now you’re a Gopher!

That’s it! Now you know a fairly decent chunk of Go. Some (pretty important) things I didn’t cover include:

• Writing unit tests, writing tests in Go is really easy! See testing
• The standard library, Go has a huge amount of useful packages in the standard library. See Standard Library.
• Goroutines and Channels, Go’s builtin concurrency is really powerful and easy to use. See Concurrency.
• Cross-Compilation, compiling your program for other architectures and operating systems is super easy. Just set the GOOS and GOARCH environment variables when building.

For more details, check the latest documentation, or for a less half-baked tutorial, please read the official Go Tutorial and A Tour of Go.