• rofl
    gv4z277ijqyn7uenr5pdvsovoawibwcjtlqgkxkfifdcs7csshpq.b32.i2p
  • rofl
    (accessible with tunnel)
  • rofl
  • rofl
    (kiwi client on webpage)
  • rofl
    Callek mostlygeek fkang mythmon hemant sphilp fuzzmz sfraser|pto relud catlee miles chartjes jlund|away wsmwk rdalal mtabara natim aselagea|buildduty jlorenzo glasserc collin5 nthomas|pto _6a68 logbot rhelmer alweezy
  • allan-silva
    hey, bhearsum, Do you think you have a time tomorrow to talk about emergency shut off ?
  • glasserc
    I guess the sentry error in Balrog stage is related to my code
  • glasserc
    I guess it's possible to have a scheduled change that isn't an insert that doesn't match any rule
  • bhearsum
  • glasserc
  • glasserc
    So yes
  • bhearsum
    i think those might be related to the prod -> stage dump we did yesterday
  • bhearsum
    i had relud run a query to remove the required signoffs, and they showed up right after that...
  • bhearsum
    so maybe it's a bug that only happens if you don't have any required signoffs
  • » bhearsum adds one
  • glasserc
    I think it's because there is a scheduled change to a required signoff that isn't there any more
  • bhearsum
    ah
  • bhearsum
    ok, so that's an invalid state
  • bhearsum
    i can probably remove those with some manual API requests...
  • glasserc
    Side note, is it just me or is it really hard to think through the implications of "stacking" scheduled changes and required signoffs?
  • bhearsum
    are you talking about how you need to fulfill the time requirement + signoff requirements before a scheduled change can be enacted?
  • glasserc
    I'm not sure.. something about it seems very recursive and mindbending to me
  • glasserc
    Like, you have to get some people to sign off on changing the act of some people signing off
  • bhearsum
    oh, yes
  • bhearsum
    that's a security thing
  • bhearsum
    the whole goal of multiple signoffs was to protect against single bad actors or compromised accounts from doing bad things
  • bhearsum
    so you have to ensure that no single person can modify stuff, or signoff on their own
  • bhearsum
    and you need to ensure they also can't _remove_ the signoff requirements by themselves
  • glasserc
    Do signoff requirements change so often that we need them to be modifiable through the database? Is this maybe a candidate for something that could be done in code?
  • bhearsum
    i can see an argument for doing it through the IAM service for sure, possibly directly in code or the app config
  • bhearsum
    there we go - i removed the scheduled changes and the exceptions are gone \o/
  • glasserc
    OK.. if we think this is a common occurrence, then I can make the code more robust
  • glasserc
    But I guess you said it was an invalid state
  • bhearsum
    yeah, i think it's okay to ignore. the only way i'm aware that we can get into this state is by messing with the db by hand
  • 203BAJ482
    balrog-web #200: building mozilla/balrog:master-36cb816f5d3270472f6f4aeb77bbfc9e5cfa68b7
  • cloudops-ansible
    balrog-admin #225: master-36cb816f5d3270472f6f4aeb77bbfc9e5cfa68b7 deployed to stage /cc relud bhearsum
  • cloudops-ansible
    balrog-web #200: mozilla/balrog:master-36cb816f5d3270472f6f4aeb77bbfc9e5cfa68b7 deployed to stage /cc relud bhearsum
  • alweezy
    bhearsum: Hello, Based on your comments on mozilla/balrog #406, does it mean that on renavigating to rules you don't need the url to point to the current pr_ch but the filter should have it?
  • bhearsum
    hi alweezy, i don't have time to answer questions today, sorry -- but if you drop something in the PR i'll reply when i can
  • alweezy
    Okay, Thanks!
  • Avedis777
    Hey, I'm back!
  • Avedis777
    Where can I find the "Add Scheduled Change for Rule" form?
  • bhearsum
    Avedis777: that's over on localhost:8080/rules/scheduled_changes
  • bhearsum
    i think it's at the top of localhost:8080/rules, too
  • Avedis777
    I mean where is the code for it?
  • bhearsum
  • Avedis777
    Is there an id for the date field?
  • Avedis777
    I want to add a condition to check if that id exists. github.com/mozilla/balrog/blob/36cb…77bbfc9e5cfa68b7/auslib/db.py#L2522 before that loop. If it does exist, I can then check if it has been changed and I can append it to the body variable.
  • Avedis777
    Am i looking at this the right way? Or is there a different approach
  • Avedis777
    Oh and im working on this bug in case anyone wants to help :) bugzilla.mozilla.org/1386400
  • bhearsum
    Avedis777: what do you mean by "id for the date field"?
  • Avedis777
    an id or a variable to reference the date in the form
  • bhearsum
    ah
  • bhearsum
    ok, so i don't think you need to think about the web form at all here - everything in db.py is at a lower level than that
  • bhearsum
    so what you're looking for instead is the database column name, which is "when" (defined at github.com/mozilla/balrog/blob/36cb…b77bbfc9e5cfa68b7/auslib/db.py#L901)
  • Avedis777
    So am I on the right track?
  • bhearsum
    you're looking in the right area of db.py, yep
  • bhearsum
    it's just important to realize that in that file, you're a bit far removed from the web layer - so you can't think of things in terms of what the HTTP request looks like
  • Avedis777
    Python is hurting my head, ive never seen or used it before
  • Avedis777
    how can I print table["time"]?
  • Avedis777
    Can I use the console to see messages that I trigger?
  • bhearsum
    Avedis777: you should be able to see anything you've printed in the docker-compose output
  • bhearsum
    Avedis777: i won't be able to reply for a bit - i'm about to start a deployment. you're on the right track though!
  • cloudops-ansible
    balrog-web #200: mozilla/balrog:master-36cb816f5d3270472f6f4aeb77bbfc9e5cfa68b7 canary deployed to prod /cc relud bhearsum
  • cloudops-ansible
    balrog-admin #225: master-36cb816f5d3270472f6f4aeb77bbfc9e5cfa68b7 deployed to prod /cc relud bhearsum
  • bhearsum
    looks like everything is OK so far, the iop spike is from some failed nightlies that had to be respun - nothing to be concerned about
  • relud
    oh, cool. glad you recognize what that's from. also, note that it's only having an effect on the replica, which is not in use in any way.
  • relud
    bhearsum: lgtm. you wanna promote now, or give it more time to stew?
  • bhearsum
    looks like the effect on the replica is neglible anyways - just a few ms
  • bhearsum
    i'm happy to proceed now - no new exceptions nor anything of note on datadog
  • relud
    k, promoting
  • cloudops-ansible
    balrog-web #200: please check balrog canary and promote to full deploy /cc relud bhearsum
  • cloudops-ansible
    balrog-web #200: mozilla/balrog:master-36cb816f5d3270472f6f4aeb77bbfc9e5cfa68b7 deployed to prod /cc relud bhearsum
  • bhearsum
    relud: the good news about these extra nightlies is that they seem to be instantly proving that the perf problems are fixed
  • relud
    nice
  • relud
    bhearsum: tags not named master-* or latest will now build stage
  • bhearsum
    awesome
  • bhearsum
    let me make a quick one by hand to give it a try
  • » bhearsum waits on stage to respond
  • bhearsum
    relud: should i expect an irc notification about the stage deploy still?
  • relud
    yes
  • relud
    I'll check on it in a moment
  • relud
    bhearsum: you have push agent too, or it won't work
  • relud
    should generally push agent first
  • bhearsum
    aaaah, okay
  • bhearsum
    will anything bad happen if i push agent afterwards?
  • » bhearsum doesn't chance it
  • relud
    bhearsum: okay, i found the bug on my side, and fixed it.
  • bhearsum
    ah, okay
  • cloudops-ansible
    balrog-admin #226: building bhearsumtest2
  • bhearsum
    sweet
  • relud
    if you push again, it should trigger a build, and you don't have to push a new tag for it to work
  • cloudops-ansible
    balrog-admin #226: failed in build step /cc relud
  • cloudops-ansible
    balrog-web #201: failed in build step /cc relud
  • relud
    ^ those are because the image pushed was not built by CI
  • cloudops-ansible
    balrog-admin #227: building bhearsumtest3
  • bhearsum
    when you say CI is that my CI or yours?
  • relud
    yours
  • bhearsum
    ahhh
  • cloudops-ansible
    balrog-admin #227: failed in build step /cc relud
  • cloudops-ansible
    balrog-web #202: failed in build step /cc relud
  • bhearsum
    okay, i'll have to wait until i land my CI changes to create version tags there to verify this, i guess
  • cloudops-ansible
    balrog-web #203: building mozilla/balrog:relud
  • cloudops-ansible
    balrog-web #203: failed in build step /cc relud
  • relud
    *sigh* retagging an image causes the check to fail
  • cloudops-ansible
    balrog-admin #228: failed in build step /cc relud
  • relud
    so it can't even be a re-tag. i'm going to ask if anyone else in my team has a solution to that :(
  • bhearsum
    hmmm, bhearsumtest3 was a new tag i thought
  • bhearsum
    or does "retag" mean tagging an existing rev?
  • relud
    it means that
  • bhearsum
    ah
  • relud
    so like: docker pull mozilla/balrog:latest && docker tag mozilla/balrog:relud && docker push mozilla/balrog:relud
  • relud
    and that results in a failure to verify, because it changes the digest
  • bhearsum
    ah, yeah - that's what i just did
  • relud
    yeah, i thought that would work v_v
  • bhearsum
    i'm going to try to get my other patches merged tomorrow - they should let us do tagging in response to github tags
  • relud
    cool. do you want me to change stage behavior until then?
  • bhearsum
    naw, we can leave it
  • bhearsum
    i'm going to step out for a bit, i'll be back before the nightlies start reporting in
  • bhearsum
    looks like we had our usual ~8 UTC load spike (from the cronjob IIRC), but everything is going well still
  • bhearsum
    and nightlies are running, probably close to an hour before we start to see repacks hammer balrog
  • cloudops-ansible
    balrog-admin #2: latestDEV FAILED
  • cloudops-ansible
    balrog-admin #3: latest DEV STARTED
  • relud
    ^ yesssss
  • bhearsum
    whee!
  • cloudops-ansible
    balrog-admin #3: latest DEV FAILED
  • relud
    *sigh* it's something tho
  • » bhearsum tries to trigger the CI based tagging
  • bhearsum
    version tagging, that is
  • cloudops-ansible
    balrog-admin #4: master-ecb6110ec6b51ce43047af6c36241397392b21b6 DEV STARTED
  • cloudops-ansible
    balrog-web #204: building mozilla/balrog:v2.41
  • cloudops-ansible
    balrog-admin #4: master-ecb6110ec6b51ce43047af6c36241397392b21b6 DEV FAILED
  • cloudops-ansible
    balrog-web #204: failed in build step /cc relud
  • cloudops-ansible
    balrog-admin #229: failed in build step /cc relud
  • bhearsum
    woo, at least we got the stage deploy cycle triggered with my tag
  • bhearsum
    i can probably force a new rev somehow to workaround the issues
  • cloudops-ansible
    balrog-web #1: master-ecb6110ec6b51ce43047af6c36241397392b21b6 DEV FAILED
  • cloudops-ansible
    balrog-web #2: latest DEV STARTED
  • relud
    ^ me
  • relud
    bhearsum: the build for v2.41 actually worked for the important thing. it was something on my side that failed intermittently
  • bhearsum
    oh sweet!
  • relud
    i think just pulling and repushing v2.41 should rebuild
  • bhearsum
    hmm, i'm not sure what that means. deleting v2.41 and then repushing it?
  • cloudops-ansible
    balrog-web #205: building mozilla/balrog:v2.41
  • » bhearsum throws pom poms around to encourage stage
  • relud
    you do this (if #205 works): docker pull mozilla/balrog:v2.41 && docker push mozilla/balrog:v2.41
  • cloudops-ansible
    balrog-web #205: failed in build step /cc relud
  • bhearsum
    oh, huh
  • relud
    *sigh*
  • relud
    docker why
  • bhearsum
    i didn't know that would do anything
  • relud
    yeah, that triggers a push event
  • cloudops-ansible
    balrog-admin #230: failed in build step /cc relud
  • relud
    but apparently it also changes digest too >:(
  • bhearsum
    hehe
  • cloudops-ansible
    balrog-admin #5: latest DEV STARTED
  • bhearsum
    do you want me to try a new tag?
  • bhearsum
    i can fire the same thing as before by creating a new Release in Github
  • cloudops-ansible
    balrog-web #2: latest DEV FAILED
  • cloudops-ansible
    balrog-admin #5: latest DEV FAILED
  • relud
    bhearsum: sure
  • relud
    is there a command to init an empty db for dev?
  • relud
    upgrade-db throws an exception
  • bhearsum
    yup, create-db should do it
  • bhearsum
    i'm pretty sure that will bring it up to the latest version too
  • bhearsum
    ok, a v2.41-real tag should come along shortly
  • relud
    cool
  • cloudops-ansible
    balrog-admin #6: master-ecb6110ec6b51ce43047af6c36241397392b21b6 DEV STARTED
  • bhearsum
    come on dev, you can do it!
  • cloudops-ansible
    balrog-admin #231: failed in build step /cc relud
  • relud
    dev web is working now: curl aus5.dev.mozaws.net/__version__ -siw '\n'
  • bhearsum
    sweet
  • cloudops-ansible
    balrog-admin #6: master-ecb6110ec6b51ce43047af6c36241397392b21b6 DEV FAILED
  • bhearsum
    ah, looks like it ended up on the wrong rev because of the new tag i created for stage
  • bhearsum
    i'll have to fix that script to not create a master-abcdef tag
  • cloudops-ansible
    balrog-web #3: master-ecb6110ec6b51ce43047af6c36241397392b21b6 DEV FAILED
  • bhearsum
    we're getting a bit of nightly traffic to prod now, not a ton yet
  • bhearsum
    everything is good so far
  • cloudops-ansible
    balrog-web #206: mozilla/balrog:v2.41-real deployed to stage /cc relud bhearsum
  • relud
    ^ yessss
  • cloudops-ansible
    balrog-web #3: master-ecb6110ec6b51ce43047af6c36241397392b21b6 DEV FAILED
  • bhearsum
    :D
  • cloudops-ansible
    balrog-web #5: master-ecb6110ec6b51ce43047af6c36241397392b21b6 DEV STARTED
  • cloudops-ansible
    balrog-web #5: master-ecb6110ec6b51ce43047af6c36241397392b21b6 DEV FAILED
  • cloudops-ansible
    balrog-web #5: master-ecb6110ec6b51ce43047af6c36241397392b21b6 DEV FAILED