• glandium
  • dmajor
    glandium: ok, thanks. bottom line is I don't think it's msvcrt
  • glandium
    dmajor: isn't int 3 a MOZ_ASSERT?
  • glandium
    or something like that
  • dmajor
    glandium: MOZ_ASSERT is int 3, but not all int 3 is MOZ_ASSERT. It was doing something weird like "test rsp, 3" which looks like maybe some compiler or library generated goop
  • dmajor
    _chkstk or something
  • dmajor
    and it didn't have the telltale static string of gMozCrashReason
  • bobowen
    glandium: ping
  • glandium
    bobowen: pong
  • bobowen
    glandium: hi, it was just over bug 1380609
  • firebot
    bugzil.la/1380609 — ASSIGNED, bobowencode⊙gmail.com — Make Win10 SDK (minimum v10.0.10586.0) required for building Firefox
  • bobowen
    glandium: I'd like to get it in for 57
  • bobowen
    glandium: thanks, I put the version in a variable because I was using it in the error message as well, should I just make the variable a string?
  • glandium
    bobowen: ah, meh, either way is fine
  • bobowen
    OK thanks
  • rillian
    jkratzer: I think the crashreporter is enabled, but you have to set MOZ_CRASHREPORTER=1 in the environment for it to do anything with unofficial builds.
  • rillian
    jkratzer: your pkg-config error just looks like the correct -dev packages aren't installed.
  • rillian
    `apt-get builddep firefox` might help
  • rillian
    or you can try going through the package list at github.com/mozilla/gecko-dev/blob/m…ython/mozboot/mozboot/debian.py#L49
  • rillian
    gandalf: groups.google.com/d/msg/mozilla.dev.platform/5srcnP-Wj4E/KvoaSUlRAgAJ is the best setup at the moment, I think.
  • rillian
    you can also just --enable-ccache=sccache but that (a) runs the preprocessor for C/C++ cache hits, which can be slower than traditional ccache, and (b) some people have reported conflicts between icecc and sccache. I haven't investigated so YMMV.
  • rillian
    as for setting -j programmatically, mystor wrote gist.github.com/mystor/27beaf86a4868a357dcd58f585b42c6a
  • rillian
    note you need to update the scheduler to match your icecream config
  • ato
    rillian: That script can’t be right. It reports 140 in London, but icecc-monitor is showing me a total of 60 at moment.
  • ato
    I thought icecc-monitor “jobs” were total number of cores for the given arhcitecture.
  • ato
    Oh the IP of the scheduler is hardcoded (-:
  • rillian
    aha :)
  • ato
    Using icecc-scheduler.corp.lon2.mozilla.com reports 80, which is still off though.
  • rillian
    surely they're both talking to the same scheduler interface?
  • ato
    Someone just switched on a new machine, so we’re up to 76 now in icecc-monitor. So I think the script includes the machines that are rejecting jobs. (Has job limit set to 0 in /etc/icecc.conf.)
  • ato
    My laptop is on the icecc network, but refuses icecc jobs, and this machines cores are included.
  • rillian
    ato: is your laptop mac? not the script adds the local cpu count unless uname is darwin
  • ato
    x86_64 Linux. The Darwin jobs appear to be excluded.
  • ato
    There’s an 8 core Darwin machine, but 60 + 4 (number of cores on my laptop) makes sense.
  • ato
    ICECC_ALLOW_REMOTE="no" or ICECC_MAX_JOBS="0".
  • rillian
    huh, vlan won't let me query lon2
  • rillian
    but at least we've finally used the number in the office domains!
  • ato
    I guess an extra 4 jobs isn’t a big issue (-:
  • rillian
    not really. Or you can delete that part of the script
  • ato
    I always use a few more jobs than I have because of poor saturation.
  • rillian
    *nod*
  • mystor
    ato: in the newest version of the script I actually use 1.4x the number of jobs I theoretically have access to for saturation
  • mystor
    And yes, it only searches for x86_64 jobs because I only built it for myself and I'm using Linux :-)
  • jkratzer
    rillian: I believe I've installed all necessary dependencies - pastebin.mozilla.org/9032300
  • rillian
    jkratzer: if you're not packaging firefox, I don't recommend using any of the in-tree mozconfig files. They assume a particular build environment which most developer systems won't reflect.
  • jkratzer
    rillian: do you happen to have an example of a debug mozconfig?
  • nalexander
    coop: gps: hey, do we have any data on what editors Mozilla engineers are using? Any chance we have (editor, contribution area)?
  • RyanVM
    (configure will actually error out if you try to use an in-tree mozconfig unless you set an env var)
  • nalexander
    coop: gps: I'm interested in what folks who write JS are using.
  • rillian
    jkratzer: I have something like:
  • rillian
    ac_add_options --enable-debug
  • rillian
    ac_add_options --enable-optimize=-Og
  • jkratzer
    RyanVM: I patched out the check in build/moz.configure/init.configure
  • jkratzer
    rillian: that's it? Do you need to specify clang paths, etc?
  • jkratzer
    or will it just use gcc if clang isn't specified
  • jkratzer
    clang/llvm
  • rillian
    jkratzer: it will use the default compiler on your system
  • jkratzer
    nalexander: I use WebStorm - we have a license for it
  • rillian
    so you only need to set CC/CXX if you want something different
  • jkratzer
    rillian: thanks - I'll give that a shot
  • nalexander
    jkratzer: ta. I'm looking for aggregate usage, though, to understand some editor integrations and where the highest value might be.
  • froydnj
    nalexander: send a google survey to mmayo-all? :)
  • jkratzer
    nalexander: ah right, my apologies
  • nalexander
    froydnj: thanks for your novel suggestion ;)
  • nalexander
    froydnj: BTW, do you have time to do the next pass on the Debian build images patch?
  • rillian
    jkratzer: also, try pkg-config --modversion freetype2
  • jkratzer
    pkg-config --modversion freetype2 -- 11.0.5
  • jkratzer
    rillian: ^
  • froydnj
    nalexander: I think I mostly understood the feedback and was going to try to apply some of it today or later this week
  • nalexander
    froydnj: ta. I'll try to get to that Pre: patch, but I need to wait for some management alignment before I commit to Android build work.
  • froydnj
    why does the number of docker image layers matter (e.g. combining RUN commands)?
  • nalexander
    froydnj: I think it's an efficiency thing; fewer layers to pull individually, faster resulting filesystems.
  • nalexander
    froydnj: but gps would know more.
  • gandalf
    rillian: thank you!
  • rillian
    jkratzer: maybe the error message is bad, and it's actually the pango bits it's missing?
  • gps
    nalexander: no clue what editors people are using
  • nalexander
    gps: it wasn't in the developer productivity surveys of yesteryear?
  • gps
    i don't think so. but i'll double check
  • gps
    nalexander: i don't see a question on editors
  • gps
    at least not one that answers what you want
  • coop
    gps: could that be added to our telemetry data?
  • gps
    coop: that's kinda hard to get. but we could try e.g. scanning process lists for known editors
  • gps
    but we'll only be able to detect what we teach it to
  • gps
    unless we send data on all running processes, which i'm not comfortable even asking for
  • gps
    froydnj: regarding docker image layer count, bug 1329282 changes everything. so i wouldn't worry about it
  • firebot
    bugzil.la/1329282 — NEW, jopsen⊙gmail.com — Deploy QEMU engine to build docker images
  • coop
    gps: could re read editor=... out of .hgrc if it exists?
  • coop
    er, we read
  • gps
    the vcs editor has a good chance of not being the primary editor
  • coop
    fair enough
  • gps
    e.g. i don't use visual studio as my vcs editor
  • gps
    there are only a finite number of editors. if we can't identify an editor for a user making commits, we could ping them
  • froydnj
    gps: just the title of that bug terrifies me
  • gps
    essentially, docker isn't very secure. building docker images in a proper vm is more secure
  • froydnj
    Imma gonna build a vm to build your vm
  • froydnj
    (I know docker's not exactly a vm)
  • gps
    vm-ception
  • froydnj
    gps: already consolidated RUN and ENV commands where I could, sooo...
  • » froydnj unsure of how to sort out this python packaging stuff
  • froydnj
    gps: my understanding of bugzilla.mozilla.org/1396098#c6 is that you're saying to put all that in a script that would pull from tooltool and run those commands, and then that script can be run during docker image creation?
  • firebot
    Bug 1396098 — NEW, nobody⊙mozilla.org — move android builds to debian-based docker image
  • gps
    having looked at debian a bit more since i did the review, are debian's built-in python packages not good enough?
  • froydnj
    I don't know; I was cargo-culting from what nalexander already had
  • froydnj
    so I assume they are not, but I don't really know
  • gps
    if you're using stretch, i think they should be good enough
  • nalexander
    froydnj: I ended up using Debian's Python packages. In your changes, you found that Debian's Python didn't support sparsecheckout.
  • nalexander
    froydnj: but for my usage, Debian's Python was good enough.
  • gps
    huh
  • gps
    stretch install pip 9.0.1, which is definitely new enough
  • froydnj
    nalexander: you mean Debian's mercurial didn't support sparsecheckout?
  • gps
    we can't use the distro's mercurial. full stop.
  • nalexander
    froydnj: huh, I meant Mercurial, not Python.
  • nalexander
    gps: oh, okay then.
  • gps
    modify install-mercurial.sh. i think i saw this in the patches.
  • gps
    actually...
  • nalexander
    gps: yeah, I think froydnj did the right thing there. I was using Debian's Mercurial.
  • gps
    i wrote what is essentially a toolchain task for building a custom mercurial .deb: hg.mozilla.org/users/gszorc_mozilla…er/system-packages/mercurial-deb.sh
  • gps
    we could upload the .deb files and install in install-mercurial.sh
  • gps
    after slightly more thought, the current patch is better
  • gps
    we'll do proper .deb packages in a follow-up
  • froydnj
    I guess we're installing non-Debian pip because python-pip pulls in a bunch of things
  • froydnj
    so sayeth the comments
  • froydnj
    python-pip on my ubuntu system has...ca-certificates, python-pip-whl, and python
  • gps
    oh right
  • gps
    yeah, it pulls in gcc and a bunch of random stuff for compiling python c extensions
  • froydnj
    that does not seem onerous
  • froydnj
    well, build-essential and whatnot (which are recommends, not requires/depends) would be a lot
  • froydnj
    but we have to install all that anyway
  • gps
    apt-get install --no-install-recommends
  • gps
    that should probably be the default for *any* apt install invocation tbh
  • froydnj
    at least, we currently are, because we're using system compilers for HOST_CC and friends
  • gps
    if we need something that is pruned out, it can be installed explicitly
  • froydnj
    we do in fact use this flag where appropriate
  • froydnj
    I guess I will try just installing the system pip and seeing where that gets us
  • gps
    tjr: i suspect you'll be interested in bug 1329282 (iirc tor uses qemu to build)
  • firebot
    bugzil.la/1329282 — NEW, jopsen⊙gmail.com — Deploy QEMU engine to build docker images
  • gps
    tl;dr taskcluster is gaining support to run QEMU
  • rillian
    gps: toolchain tasks seem to run greedily on try, regardless of whether there's another job that needs the artifact. Is that intentional?
  • froydnj
    rillian: are you modifying files that the toolchain tasks depend on?
  • gps
    rillian: they shouldn't run unless files are modified
  • gps
    the yml file defining toolchains counts :)
  • » froydnj watches builds take five minutes or more just to hit configure
  • gps
    in automation?
  • froydnj
    yes
  • gps
    where?
  • froydnj
  • gps
    single threaded tooltool download and application FTL
  • rillian
    froydnj: yes.
  • gps
    the docker and vcs bits are pretty long poles though
  • rillian
    I'm still surprised that we do this complicated overlay thing instead of just pickling everything into docker containers.
  • gps
    because bug 1291940
  • firebot
    bugzil.la/1291940 — NEW, garndt⊙mozilla.com — TaskCluster slows down considerably when using AUFS
  • gps
    plus we're using xz for the archives
  • gps
    xz is slooooow
  • gps
    faster to use lz4 and transfer more data because the network is fast in automation
  • rillian
    so...our tasks aren't actually running in docker containers?
  • froydnj
    ...do we have lz4 installed, or does tar (python or shell) understand lz4?
  • froydnj
    maybe we should just use gzip -3 or so
  • froydnj
    even the android ndk (~300MB?) took us almost a minute to download
  • rillian
    and do we make separate .xz packages for things like the clang build we download to dev machines?
  • froydnj
    5MB/s, 40Mb/s
  • gps
    we probably want xz for developers and gzip -3 or so for automation
  • gps
    tooltool does support some other archive formats
  • rillian
    froydnj: I was also sad to see the toolchain builds spending a minute or two syncing m-c just for a couple of scripts. Not that they run often.
  • gps
    but it can be touch and go depending on what tools are installed on system
  • gps
    best to stick to what python can handle natively
  • froydnj
    rewrite tooltool in rust
  • rillian
    are we not replacing tooltool with taskcluster artifacts?
  • gps
  • rillian
    "Oh, you want *that* index? Hang on while I rebuild the world..."
  • gps
  • ted
    hah
  • froydnj
    rillian: it looks like even taskcluster artifacts (e.g. sccache) get downloaded via tooltool
  • gps
    IMO use of tooltool in taskcluster represents a failure for us to build something as part of the taskgraph
  • gps
    where "tooltool" = "tooltool server"
  • gps
    do we use tooltool.py for random HTTP fetches?
  • gps
    jonas wrote some custom "download URL robustly" in python this past week too
  • rillian
    gps: I wondered about that. The logs aren't clear if it's constructing a temporary manifest, or doing something unrelated and it's just next to the tooltool downloads.
  • gps
    let's extract this to a standalone python module and do away with all this wheel reinvention
  • » gps lunches
  • rillian
    gps: requests? :-)
  • ted
    hah
  • gps
    requests doesn't do hash verification
  • gps
    but, yes, we basically need a simple wrapper around requests
  • rillian
    request_but_verify
  • rillian
    hash verification, tar/zip unpacking, output renaming, gpg signature checking.
  • rillian
    anything else on the wishlist?
  • nalexander
    rillian: command line uploading that doesn't include manifests. The tooltool upload process is counterintuitive to me.
  • nalexander
    Ability to upload from an existing (public) URL. I need to shunt the ANdroid SDK or NDK into tooltool a lot.
  • gps
    we're not writing a new tooltool client
  • gps
    just a Python API to securely download a URL
  • gps
    we could potentially also make it intelligently unpack archives too
  • gps
    so it doesn't perform decompression that isn't needed
  • gps
    although, there aren't hashes in tar files are there
  • gps
    boo
  • rillian
    nalexander: what would you like the interface of this tool we're writing to look like?
  • rillian
    the taskcluster thing isn't bad; you just write a script to leave a certain file in a certain directory.
  • rillian
    don't know if that works for private things though.
  • rillian
    sorry, this tool we're _not_ writing.
  • » rillian messed up the joke
  • nalexander
    rillian: oh, sounds like you're producing something different. Basically, I want tooltool upload URL to "just work".
  • nalexander
    mozilla/build-tooltool #39 was part of that. But let's leave it for now, tooltool sounds like it's not long for this world.
  • gps
    nalexander: write a simple shell script that does:
  • nalexander
    gps: I know, I know.
  • gps
    tooltool.py add --visibility public $1
  • nalexander
    gps: but I need to download it locally, and then upload again.
  • gps
    tooltool.py upload --authentication-file /path --message $2
  • nalexander
    And my upload bandwidth is very limited.
  • rillian
    nalexander: I usually ssh into the office and run uploads from there.
  • nalexander
    rillian: I used to ssh into people, but it's gone. What host do you have access to?
  • rillian
    still limited, but much better than at home.
  • rillian
    nalexander: I have a desktop machine in the office.
  • rillian
    I can make you an account if you'd like.
  • rillian
    or you could have my old one..
  • nalexander
    rillian: oh, right. Would you? Username nalexander
  • nalexander
    rillian: I'll send you my public key.
  • » froydnj realizes his uploads could have gone so much faster with people.m.o
  • froydnj
    missed opportunity, there
  • gps
    ssh.mozilla.com still exists IIRC
  • nalexander
    Like the old days, frantically searchign for leech shells :)
  • nalexander
    on T1 lines, when it mattered :)
  • nalexander
    gps: I thought that didn't let you run a shell?
  • nalexander
    Or is that hg.m?
  • rillian
    gps: I can't log into it
  • gps
    i have no clue. i've used it as a jumphost in the past
  • rillian
    there's jump1.community.scl3.mozilla.com or something like that
  • gps
    i also have gregoryszorc.com on some ec2 micro instance or something
  • gps
    if i need fast bandwidth, i ssh into that :)
  • gps
    super cheap
  • rillian
    yes. since they took away people I've just expensed vps rentals
  • gps
    i am using gregoryszorc.com as my IRC client. maybe i could get that expensed :)
  • gps
    i'm sure someone will tell me i should use IRC Cloud
  • gps
    ... which i don't have permission to use for some weird reason
  • gps
    i don't care enough to address that :)
  • rillian
    gps: it costs money either way. go for whichever works.
  • froydnj
    "remote: waiting for lock on working directory of /repo/hg/mozilla/try held by process '29707' on host 'hgssh4.dmz.scl3.mozilla.com/effffffc'" :(
  • froydnj
    been a while since seeing a message like that
  • » RyanVM wonders who's holding up the line
  • RyanVM
    don't they know we're wealthy businessmen who need our Try runs now?!
  • froydnj
    moar docker changes \o/
  • froydnj
    argh, the clang build has suddenly decided it has link errors on x86
  • RyanVM
    you trying to make hazard builds work with 4.0 or something?
  • RyanVM
    no, that'd be sfink
  • RyanVM
    nvm me
  • froydnj
  • RyanVM
    oooooo
  • froydnj
  • froydnj
    whyyyy
  • RyanVM
    ignorant me wants to know what that buys us
  • RyanVM
    besides getting us closer to one less compiler to have to support
  • froydnj
    well, the ability to keep compiling firefox for android when gcc is really truly dropped from the android ndk
  • gps
    android upstream moved to clang
  • froydnj
    slightly smaller binaries
  • RyanVM
    aha!
  • froydnj
    possibly faster code (haven't tested yet)
  • RyanVM
    well, that's a pretty good motivation :D
  • froydnj
    oh, and the ability to move to C++14
  • froydnj
    static analysis on android is also a possibility
  • froydnj
    which opens up other interesting things
  • froydnj
    like faster debug tests
  • RyanVM
    neato
  • froydnj
    aha, lth is my nemesis
  • gps
  • gps
    it's little things like symlink_to() being named that so you don't have to remember what direction the symlink is in
  • ted
    0:39.11 error: environment variable `MOZ_TOPOBJDIR` not defined
  • ted
    0:39.12 --> C:\build\mozilla-central\xpcom\rust\nserror\src\lib.rs:59:22
  • ted
    uggh
  • ted
    actually shit, building rust code is going to be a PITA in WSL ;-(
  • froydnj
    argh, our configury doesn't detect that x86 android with clang actually needs -latomic
  • froydnj
    fantastic
  • » froydnj decides to put that off until tomorrow
  • ted
    froydnj: what would you think about wrapping cargo invocation in a python script?
  • ted
    we have a lot of makefile goop in rules.mk around it right now
  • ted
    we could do that in python, or make a bunch of it configure-substed or whatever
  • froydnj
    ted: ooo, that's not a bad idea
  • froydnj
    I'd rather write python conditionals for all our cargo nonsense than makefile conditionals
  • ted
    yeah
  • ted
    i'm probably going to need to do the same thing for cargo i did for midl.exe: shell out to cmd.exe to run a batch script so i can set windows env vars
  • froydnj
    probably makes passing around environment variables and quoting and whatnot more straightforward too
  • ted
    yeah
  • ted
  • ted
    i don't love that, but there's currently no way (AFAIK) to pass environment variables to a windows process when running it from inside WSL
  • ted
    so my workaround is to put the env vars into a batch file, run cmd.exe on it, and then run the command from in there
  • ted
    froydnj: OK, i'll work up a patch
  • » ted needs to untangle the 50 other changes in his working copy
  • ted
    also i need to go start making dinner
  • froydnj
    so, uh, this whole process-launching overhead win that we thought we might see isn't happening, eh?
  • ted
    froydnj: it's not *that* bad
  • ted
    not everything needs this
  • ted
    we can run cl.exe just fine, and other stuff like that
  • ted
    it's just things where we need to communicate info via env vars, and there's no other way
  • ted
    midl.exe needs to find cl.exe to run the preprocessor, but we can't set PATH for it
  • ted
    it has a /cpp_cmd argument but if you pass a path with spaces something breaks
  • ted
    most of the build is just running linux binaries, so there's no fiddling at all
  • ted
    there's only a handful of midl.exe invocations
  • ted
    and cargo only gets invoked a few times
  • ted
    once i get the whole build working we can see what the timing looks like
  • ted
    configure is ~10s faster, i think
  • nemodon
    How do I build a code coverage build without crashing on asserts? I tried xpcom_debug_break env variable as warn, but that doesn't seem to work.