• travis-ci
    build-puppet#2058 (production - 6b70a3f : Ben Hearsum): The build passed. (travis-ci.org/mozilla/build-puppet/builds/286486354)
  • travis-ci
    build-puppet#2059 (master - 3e3b623 : Ben Hearsum): The build passed. (travis-ci.org/mozilla/build-puppet/builds/286486356)
  • Pike
    Callek: with the beta 5 problem, I was reminded that we have failed jobs in l10n nightlies on TC most of the time, I guess. Do you have that on the radar, and does that impact release automation on TC?
  • Pike
    also, how far are we release automation off of buildbot?
  • Callek
    Pike: by failed jobs do you mean the actual repacks, or the balrog/beetmover submission?
  • Callek
    Pike: if actual repacks I don't have that on my radar
  • Pike
    most of beetmovers, but also straight Nxx jobs on treeherder.mozilla.org/#/jobs?repo=…mchange=c3b7759671deae73e40ebca01d7
  • Pike
    f23a326a4b8c2
  • Callek
    Pike: as far as "off buildbot", I'd defer to aki or catlee atm, but we have a pretty strong "must be off buildbot by 59-goes-to-beta" since 59 is going to be esr
  • catlee
    Pike: I expect sometime in the 58 beta cycle
  • Callek
    Pike: I don't see f23a326a4b8c2 on that link
  • catlee
    l10n jobs are going to be one of the first things to move over
  • catlee
    the beetmover jobs are on our radar
  • Pike
    Callek: that's a trucated linked leftover
  • catlee
  • firebot
    Bug 1398800 — NEW, nobody⊙mozilla.org — Intermittent beetmover failure exit code: -11
  • Callek
    Pike: ahhh, let me try again
  • Callek
    Pike: so one of the N#'s I see is public-artifacts.taskcluster.net/V5…Wdyw/0/public/logs/live_backing.log which is an intermittent network issue...
  • Pike
    mzl.la/2zffDui is a revision with failures
  • Callek
    Pike: I think it won't be strongly impactful to releases, since we'll have stuff (or humans) to double check that all locales finish.
  • Callek
    Pike: I want to eventually do something similar for nightlies, so we get stronger alerts if everything isn't complete
  • Pike
    depends on how granular we can retry then, with eo/en-ZA it was "we'd have to do a full respin", and if we have some glitch in each run, we'll never succeed.
  • Callek
    Pike: that was in part because it was on Buildbot... in TC the jobs are more automic
  • Callek
    there was possibly a way we could have achieved eo/en-ZA without a full respin, but it would have been much riskier too
  • Callek
    once its moved off buildbot then the repack itself is independent of signing, and is independant of pushing those files anywhere, in buildbot much of that happens all at once, which means sha's, s3 file location caching, etc all come into play.
  • Callek
    Pike: unrelated, how goes the reworking of that patch to cleanup the l10n makefile stuff? (the patch that broke BB windows installers)
  • Pike
    I didn't have time to dive into it lately. I could probably hammer something, but I have little understanding of how rude that would be
  • Pike
    also, we have tons of bugs flying by right now to convert pieces of makefile into python actions
  • Callek
    yea, I saw some discussion yesterday in #build around just that
  • Callek
    :-)
  • Pike
    another context is bug 1387485 , which makes a couple of assertions that may or may not affect automation. Might be good to get one of you to share your perspective there
  • firebot
    bugzil.la/1387485 — NEW, nobody⊙mozilla.org — L10n repacks don't work on artifact builds
  • Callek
    ah yea, I've been eying it
  • » Callek recalls now that something in there in the last day or two caught my eye...
  • Pike
    the part of requiring every bit of l10n being passed in to configure, likely. I have a couple of opinions on that from a developer perspective, but that also affects automation. Both the code, and probably also wall-clock times
  • Callek
    Yea, I just commented, I want to get to a point where l10n doesn't require configure being run at all
  • Callek
    but if it does, that we certainly don't need to pass the ab-cd itself down through configure
  • Callek
    I'd like (pseudocode) ./mach l10n-repack ab-cd[:hg-rev] [ab-cd:hg-rev, ...] [--no-merge-locale] to do everything we need for a repack
  • Callek
    give or take some fudging
  • Callek
    that in the end allows it to work in an artifact build way, (fetches en-US binaries from automation), could allow it to work with local l10n patches (if desired), or with a specific upstream build (a try rev's build, a release job, etc)
  • Callek
    the idea in my head is that mozharness could no longer be needed even for automation (but that will be further out)
  • bhearsum
    is it possible to kill all tasks for a push on a project branch?
  • Callek
    bhearsum: yes, but we can't do it, sheriffs can (we might consider asking #treeherder folk for that access)
  • Callek
    bhearsum: I've usually pinged RyanVM to see if he's free to do so, providing him a treeherder link to the rev
  • bhearsum
    ah, thanks
  • » RyanVM ducks
  • Callek
    bhearsum: alternatively we can probably use taskcluster clients cancel support (I've never tried this): `taskcluster group cancel ...`
  • bhearsum
    i think i managed to kill all of the jobs i needed to, i probably don't need a full kill at this point - it's always seemed useful though, esp. on project branches where you're throwing things at the wall
  • bhearsum
    oh cool
  • bhearsum
    i think it kindof works, except it runs out of file handles on large task graphs
  • bhearsum
    looks like it actually retries - i think it killed everything \o/
  • RyanVM
    we allow "cancel all jobs" for all users on Try
  • RyanVM
    we could probably extend that to project branches without much trouble
  • RyanVM
    seems like a reasonable ask
  • jryans
    Sylvestre: not sure i totally follow what emails would go to release-drivers vs. release-signoff... where would QA feature signoffs go, for example?
  • jcristau
    (probably better discuss this in #release-drivers) they would go to -signoff, AIUI
  • Sylvestre
    jryans, just like jcj said
  • jryans
    ok, thanks!
  • marcia
    I haven't seen updates to today's nightly build on mac, and builds are missing on the crash dashboard. Is there an issue with today's nightly builds?
  • aki
    nightlies appear to be green. checking balrog
  • jcristau
    likewise linux
  • aki
    hm, balrog tasks haven't run
  • aki
    because beetmover hasn't run
  • aki
    because the funsize docker image task failed
  • aki
    rerunning that, which should unblock the rest of the graph
  • aki
    timed out after 2hrs, and it hadn't finished setting up yet
  • bhearsum
    so i ended up getting lost in a maze of taskcluster transforms and scopes when i tried to run some nightlies on jamun today to verify my balrog patches. i ended up aborting everything when i realized that jamun seems to have a bunch of level 3 permissions
  • bhearsum
    it looks like that was done when we were doing the initial Dawn work...which i think was because we needed real certs to verify various update paths
  • bhearsum
    does anybody know if that's still needed? it seems like we probably shouldn't be signing with real certs for staging releases...
  • aki
    staging releases in bb require release signing
  • bhearsum
    huh, why?
  • aki
    i'm trying to do dep signing releases in tc
  • aki
    probably mars + update testing require release sigs
  • bhearsum
    yeah, update verify is likely to get grumpy about signatures withotu proper sigs. i think that's something we've just accepted in the past
  • aki
    it puts a damper in end to end testing if you have known failures -- unknown if the failure is hiding a real failure
  • aki
    so ideally in dep staging releases, we'll have green update verify with dep signing somehow
  • bhearsum
    have we decided to live with the risk of having properly signed MARs that we actually don't want users to have?
  • aki
    might involve a staging flag
  • aki
    we release-sign beta on push, so we have a ton of release signed things we don't want users to have
  • bhearsum
    ah
  • aki
    i've been suggesting we sign with the dep key and sign release when we know we want a build promoted, but that's way futured
  • aki
    doable in TC
  • bhearsum
    tricky with windows or OS X, where you're touching internal bits that could potentially break the build after you've tested it
  • aki
    yeah, various people were working on tools to verify only the signature changed between binaries
  • bhearsum
    we used to do that way back in the 2.0 era, and that was part of the reason we started doing proper signing up front
  • bhearsum
    ahh, that's cool
  • bhearsum
    even the signature change can break stuff (eg: verification of updater.exe by the maintenance service), but it would reduce risk a lot
  • aki
    yeah. ideally we have a small suite of tests that run post-re-sign
  • aki
    that check for those things
  • bhearsum
    alright - so coming back to my original thing about jamun at level 3 - it sounds like that's desireable right now?
  • aki
    it's something we're living with, because we can't fix it yet :)
  • bhearsum
    ok
  • bhearsum
    so release signing, but we'll submit to balrog stage
  • aki
    it is good to make sure you're not pushing to prod beetmover or balrog; i believe that's the case, since that's controlled on the bb side on jamun
  • aki
    er, nightlies may push to prod balrog with nightly-jamun, not sure
  • bhearsum
    i think they do right now
  • bhearsum
    because that's the only one that works the vpn proxy AFAIK
  • aki
  • aki
    appears to not specify jamun, which i think means we go to dep hg.mozilla.org/projects/jamun/file/…taskgraph/util/scriptworker.py#l162
  • bhearsum
    right
  • aki
    so unless something else is hacked, we should be going to balrog-stage
  • bhearsum
    i think nightlies were disabled until i enabled them a few hours ago, so it's tough to be sure
  • bhearsum
    but maybe we don't care about nightlies for staging release purposes
  • aki
    we do, because jamun release-signs nightlies to promote
  • aki
    nightlies on beta == promotable build
  • bhearsum
    oh right
  • bhearsum
    so it's actually confusing
  • aki
    but i imagine you can test nightly stuff, and then back out to allow for staging betas
  • bhearsum
    because those "nightlies" on jamun are created by the on-push decision task
  • aki
    yes
  • aki
    really they should be "shippable"
  • bhearsum
    yeah, okay
  • bhearsum
    right, so we don't care about cron-triggered nigthlies there
  • bhearsum
    we care about the shippable builds, and then all of the stuff that the release graph does
  • bhearsum
    ie: we can probably ignore the in-tree balrog submission for the moment
  • aki
    yes. the shippable on-push "nightlies" use the same config as the cron nightlies, though
  • bhearsum
    ah
  • aki
    they just weed out the balrog/beetmover tasks
  • bhearsum
    so that matters for signing, but not for balrog
  • aki
    yes
  • bhearsum
    clear as mud!
  • bhearsum
    i do think i understand a bit better now - thanks for all the knowledge transfer
  • aki
    np. sorry shippable builds are confusing and conflating "nightly"
  • aki
    i think we're aiming for q1 to actually implement shippable builds properly
  • bhearsum
    no worries
  • bhearsum
    what does "properly" mean?
  • aki
    call them "shippable" instead of "nightly"; have nightly promotion; get rid of "pgo" builds in favor of pgo-enabled shippable builds; etc
  • bhearsum
    ah
  • aki
  • bhearsum
    is this independent of in-tree release graphs?
  • aki
    related
  • bhearsum
    but not necessarily interdependent?
  • aki
    in-tree release graphs will ideally happen this quarter, and shippable builds will bring promotion to nightlies
  • aki
    in q1, possibly
  • aki
    optimistically maybe
  • bhearsum
    thank
  • bhearsum
    s
  • travis-ci
    build-puppet#2061 (production - 408968d : Aki Sasaki): The build passed. (travis-ci.org/mozilla/build-puppet/builds/286628381)
  • sfink
    how does one use an osx build loaner? I need to compile stuff. It says I need to run mach bootstrap. mach bootstrap wants xcode. So it sends me to the app store for it, I request the install, and it says I need OSX 10.12.6. I have 10.7.2 on the loaner.
  • sfink
    is xcode lurking somewhere on the system already?
  • sfink
    ooh, my locate db finished and pointed me to /Developer/Applications/Xcode.app!
  • sfink
    sadly, doing xcode-select to swtich to it doesn't seem to convince mach bootstrap that I still need to install it :(
  • aki
    almost all of our osx builds happen on linux these days. did you need to compile non-gecko?
  • sfink
    yes. I'm debugging build problems for the standalone spidermonkey release.
  • sfink
    so it needs to be native osx; I already have it fixed up for linux.
  • aki
    ok
  • aki
  • sfink
    I'm stepping through mach bootstrap now
  • sfink
    I think it might be something dumb
  • aki
    oh, you said that
  • travis-ci
    build-buildbot-configs#2946 (master - b809cb5 : Aki Sasaki): The build was broken. (travis-ci.org/mozilla-releng/build-buildbot-configs/builds/286747350)
  • travis-ci
    build-buildbot-configs#2947 (production - 89b816b : Aki Sasaki): The build was broken. (travis-ci.org/mozilla-releng/build-buildbot-configs/builds/286747373)
  • travis-ci
    build-buildbot-configs#2948 (master - 91ec718 : Aki Sasaki): The build was fixed. (travis-ci.org/mozilla-releng/build-buildbot-configs/builds/286752223)
  • travis-ci
    build-tools#2031 (master - f5372ae : Tom Prince): The build passed. (travis-ci.org/mozilla/build-tools/builds/286752173)
  • travis-ci
    build-buildbot-configs#2949 (production - 0163f1b : Aki Sasaki): The build was fixed. (travis-ci.org/mozilla-releng/build-buildbot-configs/builds/286757954)