• Dexter
    trink, looks like you forgot to r+ mozilla-services/mozilla-pipeline-schemas #88 yesterday
  • trink
    Dexter: you already have my feedback and discussion in IRC
  • trink
    I am not the gate keeper on that repo
  • Dexter
    ok
  • trink
    someone also has to take on tagging and packaging releases
  • trink
    or automate it to push the RPM back to github on the tagging of master
  • Dexter
    frank, sunahsuh any suggestion on how to mock an http request in a scala test? I'm planning to add a test in telemetry-batch-view
  • frank
    Dexter: I actually just did that
  • sunahsuh
    Dexter: which resource?
  • Dexter
    \o/
  • frank
  • Dexter
    I'm adding it for AMODatabase/AddonRecommender :)
  • frank
    it uses WireMock
  • frank
    Dexter: this simplified test is probably the easiest to parse: github.com/fbertsch/telemetry-strea…ing/TestEventsToAmplitude.scala#L53
  • sunahsuh
    i'd also consider whether you can replace whatever method fetches the request (like we do for tests to replace the histogram/scalar tests)
  • sunahsuh
    er, words. not enough coffee.
  • sunahsuh
    frank: are those packages only used for tests? if so, i think we should consider building test vs prod fat jars
  • frank
    sunahsuh: yes definitely, luckily telemetry-streaming is thinner than t-b-v but definitely something to put on the radar
  • sunahsuh
    ohh, didn't notice the repo
  • Dexter
    thanks Sunah, Frank
  • Dexter
    given that I'm on t-b-v, is that ok to add the wiremock dependency or should I go with sunahsuh 's approach? :)
  • chutten
    robotblake: "Query in queue" cluster overview says there's nothing in the queue
  • frank
    Dexter: I personally think we should try and push towards not making special code paths for tests, I think having this example with wiremock would be great
  • frank
    I tried to mock the endpoint when I added those local histograms and scalars, but couldn't get it to work initially
  • Dexter
    super, thanks !
  • sunahsuh
    frank and I have different testing philosophies, I think :) if you're doing more unit-testy things, replacing code for focused tests is fine, but we don't really write unit tests :x
  • Dexter
    I think both approaches are valuable, while leaning more towards using specialized mocking libs to do this kind of stuff
  • natim
    Do you know what is the format of schedule_until in STMO?
  • sunahsuh
    natim: there are a few more stmo experts in #datatools :)
  • natim
    I tried using the form schedule until date (with the format: 2017-11-02T00:00:00.000) but in the record itself I still see schedule_until: null
  • natim
    It might be a bug
  • chutten
    Query's executing now
  • frank
    mreid: why the heck am I getting a 404 on mozilla/firefox-data-docs #58
  • frank
    wait, the link is originally /pull/58, then redirects to /issues/58...
  • mreid
    that PR disappeared...
  • frank
    mreid: did you remove it?
  • mreid
    no
  • frank
    wutt
  • mreid
    wth
  • frank
    mreid: for a split second it said "3" again next to the Pull Requests tab
  • frank
    but now... gone
  • mreid
    uh
  • mreid
    I can re-submit
  • frank
    hah! I suppose you should
  • frank
    mreid: it just showed up in the list of PRs
  • frank
    but I can't view it
  • frank
    well, gone again
  • sunahsuh
  • sunahsuh
  • mreid
    sunahsuh++
  • mreid
    I shall just wait
  • frank
    ah, that would explain the inconsistency
  • mreid
    heh, now your approval is not showing up :)
  • frank
    mreid: I will "eventually" approve, I promise :)
  • su
    here: does anyone know how to access crash-pings? is it via the Dataset API?
  • su
    in ATMO
  • chutten
    su: Should be doctype="crash"
  • chutten
    And yeah, Dataset
  • su
    chutten: awesome, thanks so much! :)
  • chutten
  • su
    ohhh~ thank you!
  • mreid
    there's also a "crash_summary" table in parquet
  • mreid
    su: ^
  • mreid
    rrayborn: as possibly the only known user of the HBaseMainSummary API in python_moztelemetry, are you ok with us removing it?
  • mreid
  • firebot
    Bug 1402322 — NEW, nobody⊙mozilla.org — Consider Removing HBase Main Summary
  • mreid
    huh
  • mreid
    running a query against Athena:
  • mreid
    Error running query: Query exhausted resources at this scale factor
  • frank
    robotblake^
  • robotblake
    Which query?
  • frank
  • frank
    looks like it may be transient
  • jgaunt
    frank, sunahsuh - got introduced to a new pyspark error today:
  • jgaunt
    ValueError: Cannot convert column into bool: please use '&' for 'and', '|' for 'or', '~' for 'not' when building DataFrame boolean expressions.
  • jgaunt
    I'm puzzled because the df works well enough to export completely as CSV
  • jgaunt
  • jgaunt
    the error is in [12]
  • jgaunt
    have either of you run across this before?
  • jgaunt
    if not I'd wager it's more a pandas problem than spark
  • jgaunt
    pls lmk