Is homu having a bad day? buildbot2.rust-lang.org/homu/queue/rust shows nothing pending (an of course sereval approved)

uh, the last commit in the auto branch is 7 hours old...

ping acrichto: homu seems stuck

acrichto, bors is still stuck...

I think it’s still early morning in his time zone

uh, yeah, timezones are fun

anybody else can restart homu?

I don't think so, I think he's the only infrastructure member with full permissions

:/ let me see if clicking that "Synchronize" button will fix it

"Synchronizing rust..."

(populating...)

nope doesn't work \o/

(try to recreate the auto branch)

(doesn't matter either)

looking into bors

apparently cargo needed synchronizing?

and then kicking it worked?

no idea

working now 🎉 cargo submodule? but that PR is merged quite long time ago

kennytm: er the cargo project needed synchronizing

homu was tripping some assertion somewhere

and re-sync'ing the cargo queue appears to have fixed that

and then everything started to proceed as usual

oh...

:/

I know it's boring, but I'm seeking comments on internals.rust-lang.org/t/rfc-build…elf-hosted-ci-infrastructure/7085/1

ishitatsuyuki: We only have 2 cores on Travis

Zoxc: no, it's 4

Travis logs prints cpu cores : 2

An undocumented machine type

(it's referenced in source, search n1-standard-4)

Really? Last time I checked it it was indeed quad

(checked again, indeed quad core)

we have 4 cores for Linux, 3 cores for macOS, 2 cores for Windows (which should have been upgraded to 4 cores in mid-Feb 😂)

lol, don't count on AppVeyor

ishitatsuyuki: Show me the logs? =P

Zoxc: travis-ci.org/rust-lang/rust, click on any log, search for the cpuinfo

I don't find anything. Searching for GenuineIntel gives me 2 cores

is this a "cores versus hyperthreads" thing? >_>

lol

if HT doesn't count we should consider AppVeyor having 1 core 🙃

alternately, is the environment different between the builder every PR gets and the builders that bors uses?

the builders should be configured similarly

in api.travis-ci.org/v3/job/356147819/log.txt i see four "processor info" blocks, each saying "cpu cores: 2"

this is a bors builder

I wonder if you get 1 core if you get an EC2 instance with 2 vCPUs

maybe we should say "4 cores" means we pass `-j4` when building stuff

and that "4" is whatever number reported by the num_cpu crate

That is not what cores mean =P

DigitalOcean sells dedicated hyper-threads. Hyper-threads are by definition shared...

naming is hard :p

Just call them hyper-threads =P

OVH rents out dedicated consumer CPUs <3

Overclocked consumer CPUs even

An ordinary OVH dedicated had hyperthreads just as same as other providers

ishitatsuyuki: I don't know what you mean by that

I mean it doesn't seem to rent dedicated (non-HT) cores

They rent dedicated servers =P

ovh.com/world/dedicated-servers/game/171mc4.xml 5 Ghz turbo

Uh well, they clearly state they're using HT

But otherwise it's just as same as other providers

Moving back to the original topic, how useful do you think managing our own infrastructure is?

ishitatsuyuki: And why is using HT a problem?

Zoxc: it isn't, but not clearly stating that can lead to wrong assumptions

I think using our own infrastructure is useful as we can get faster machines

Having an "unlimited" and fast cache would be useful too

ishitatsuyuki, there was a meeting about that a few weeks ago, acrichto and aidanhs weren't too happy to manage more infra though

Hmm, I see the point. Though, I think the situation may improve if we introduce automation/orchestration, aka self healing

Nuke everything when the queue is empty =P

Zoxc, did we ever have an empty queue? :o

Yes

Delete the rust-lang/rust repo, start from OCaml

ishitatsuyuki: That requires a bunch of work. Firefox has done something like that, and they have had several people working on it full time for multiple years.

tomprince: i keep hoping we can get rust/servo onto taskcluster and leverage the firefox investment there

if you can share the work mozilla has done for firefox that would be great

ted: what's the status of platform support with taskcluster?

I think it supports all three

But I'm not sure if TC is flexible enough to support our GitHub workflow

It definitely is flexible enough, although if it doesn't match the existing workflow support, there'd need to be some code written to interface with it. Still, easier that building our own platform by several orders of magnitude, probably.

i don't know what the mac support actually looks like, since we only use it for tests with firefox

but the worker implementations are pretty good nowadays, it's definitely possible to throw hardware at the problem

I think deepspeach and nss both might have some mac builders?

that sounds pretty likely

Provisioner localprovisioner

WorkerType nss-macos-10-12

ted: what does a job get executed in? VM or container?

ishitatsuyuki: i don't know what the mac ones are

linux workers are docker containers in EC2 instances

windows workers are in EC2 instances but they don't start a fresh instance per task

Do they share resources?

That do create a fresh user, I think, though.

the worker does some sandboxing etc, it creates a new user per task

yeah

ishitatsuyuki: i'm only really familar with the AWS provisioner, which lets you define worker types with a "capacity" for how many simultaneous jobs can run on the worker

for most of our firefox workers it's just set to 1

There is also (unfinished, as yet) work on a kvm/qemu worker.

right

one nice thing about taskcluster is that it actually does have a nice well-thought out architecture

so things like implementing new worker types is feasible

In the current model we're constantly spinning a fixed amount of builders so EC2 doesn't really sound cost effective though

i dunno! firefox CI is very bursty, so being able to ramp it up and down is great

and TC uses spot instances which are cheaper

and EC2 has per-second billing now, so it can turn off instances when they're not needed

having ephemeral workers has other nice benefits though, like when amazon announced the new c5 instance types we were able to get tasks running on them by doing a little work to get the worker AMI working and then just adjusting worker type definitions

a lot of the things taskcluster made better for firefox CI are things that you've already got in rust, like having most of your CI defined in-tree

acrichto: aidanhs: possibly worth talking about ^ at the all hands next week

there's also something to be said for having coworkers that work on your CI tooling

aturon: indeed! although I think it'd be most productive with a taskcluster expert we can grill in real time

which may mean we should discuss it mostly at a high level and then get ted on a call afterwards

acrichto: i agree, but it also might be worth spending a bit of time thinking of what info we'd need to know and getting some answers in advance

oh of course yeah

we just gotta be real explicit

b/c otherwise that convo would derail quickly...

i'm not a taskcluster expert, i just play one on TV :)

we have some actual taskcluster experts we could rope in, and i am always happy to facilitate

but yeah, if you're going to have the conversation you want to have done some homework

what things are important to you, what you're looking to gain

Maybe we can work on homu support for it first then?

i wonder if you can reuse (or get anything useful) from the conduit stuff people are working on for firefox? wiki.mozilla.org/EngineeringProductivity/Projects/Conduit

conduit encompasses homu-like functionality, but integrated with phabricator for code review as well

Well, conduit doesn't have the interaction with CI bit at all.

Just landing code, which homu also has.

I think homu just works by pushing to specific branch, which gets picked up by CI, and then looks at status.

All of which tc supports natively.

Yeah, but in homu the status needs some coding

In bors-ng it's a matter of one line config

Same in homu: github.com/rust-lang/rust-central-s…blob/master/homu.toml.template#L159

Oh, looks like you're right

ishitatsuyuki: WebRender has been using TaskCluster with Homu

though I don’t know how much Servo’s fork of homu has diverged

Awesome.

SimonSapin: from github.com/barosl/homu , servos is very different, but rust is using servos forck

ah ok

Is it likely that we'll be start the work on TC soon? Can some team member create a thread to discuss this?

Thanks for the thread, I'm very excited to be a part of the movement

i suspect there's some politics involved in that decision :)

kennytm: 'extreme bors planning' means "figuring out what we want to do to fix bors (mainly cycle time), both short and long term"

aidanhs: i see

it's a tongue in cheek comparison to this being an 'extreme sport' but it's not very clear - will change

clearly we need to paint flames on the sides of the bors logo

heh

get mountain dew to sponsor rust development

maybe I should get a crate and slam it down in the first all-hands infra meeting to get us going

ted: you joke, and then i started daydreaming about what it would take to court pepsico to start rustifying some logistics software or something

haha

today: rust in small places in logistics pipeline; tomorrow: whitepaper on how efficient they got with rust; soon: rust gamedev summit sponsored by mountain dew

someone please review {!gh 49246}, it is likely needed to unblock the bors queue 🙏