SQR-027: Getting SQuaSH metrics to Honeycomb

  • Angelo Fausti

Latest Revision: 2018-09-21

Getting started

Check the quick start tutorial to learn about Honeycomb UI capabilities. Then join the LSST SQuaRE team using this invitation, the datasets created in this notebook are shared with the team.

What is observability?

The Honeycomb observability manifesto is a good place to start. Also check the introduction to observability here.

NOTE: SQuaSH metrics are high level metrics, they are not the kind of ‘value’ that one would want for observability. However, in this notebook we’ll send them to Honeycomb just because they are the only data that we have in SQuaSH for now. That’s still useful to illustrate the use of libhoney and beeline-python integrations.

In my opinion, the benefit of this tool will become clearer by sending more context data from our DM pipeline tasks, and ultimately as an analysis tool for the Engineering and Facilities Database (EFD).

Events and Datasets

Honeycomb data is a series of Events, each of which represents something in your environment worth tracking. When you send events to Honeycomb, you collect related or recurring events under a single Dataset.

An Event can be anything:

  • an HTTP request to your app
  • An SQL query
  • A job is submitted to a queue
  • An execution of a LSST DM pipeline task
  • A detection of particular astronomical object

Datasets are used to partition your data into separate and queryable sets. See best practices for defining datasets in Honeycomb.

Let’s start by identifying a validate_drp run as the Event containing the metrics measured by each run.

For illustration purposes we’ll collect those metrics from the SQuaSH API:

In [1]:
SQUASH_API_URL = "https://squash-restful-api-demo.lsst.codes/"
In [2]:
import requests
r = requests.get(SQUASH_API_URL + "/job/1").json()


data = {'id': r['id'],
        'date_created': r['date_created'],
        'filter_name': r['meta']['filter_name'],
        'dataset': r['ci_dataset']}

# events should be flat dict
for meas in r['measurements']:
    data[meas['metric']] = meas['value']

data
Out[2]:
{'id': 1,
 'date_created': '2016-09-15T00:00:00Z',
 'filter_name': 'r',
 'dataset': 'validation_data_cfht',
 'validate_drp.AM1': 4.0,
 'validate_drp.AM2': 5.0,
 'validate_drp.PA1': 7.0}

This looks like an structured log entry, it also looks like a property set or it may recall something else...

Sending events to Honeycomb

  1. When honeycomb gets a request to add an event to a dataset that doesn’t exist yet, it creates the dataset.
  2. The dataset schema is inferred automatically. It also will infer automatically if you add new fields to your events.
  3. Events are added to the dataset for querying.

NOTE: In order to execute this cell you will need the HONEY_API_KEY. It is available from “Team Settings” in the UI once you join the lsst-square team.

In [3]:
import libhoney

HONEY_API_KEY = ""  # Obtained from the UI under "Team Settings"

libhoney.init(writekey=HONEY_API_KEY, dataset="single-event-demo")
libhoney.send_now(data)

That’s it! Check the new dataset at https://ui.honeycomb.io/lsst-square/datasets

NOTE: this demo dataset may already exist from previous executions of this notebook, make sure you delete it before running the notebook if you want to start from an empty dataset. If you don’t, note that they will be inserted in the existing dataset.

Now, let’s send all squash data to Honeycomb

Actually, just the scalar metrics and some context information.

Create the squash-demo dataset:

In [4]:
libhoney.init(writekey=HONEY_API_KEY, dataset="squash-demo")
builder = libhoney.Builder()

Now loop over all jobs:

In [5]:
from datetime import datetime

jobs = requests.get(SQUASH_API_URL + "/jobs").json()

for job_id in jobs['ids']:

    r = requests.get(SQUASH_API_URL + "/job/{}".format(job_id)).json()

    if r['ci_dataset'] == 'unknown' or r['ci_dataset'] == 'decam':
        continue

    print('Sending event for job {}...'.format(job_id))


    # Spawn a new event and override the timestamp
    event = builder.new_event()
    event.add_field('id', job_id)
    event.add_field('filter_name', r['meta']['filter_name'])
    event.add_field('dataset', r['ci_dataset'])

    for meas in r['measurements']:
        event.add_field(meas['metric'], meas['value'])

    event.created_at = datetime.strptime(r['date_created'], "%Y-%m-%dT%H:%M:%SZ")

    event.send()
Sending event for job 1...
Sending event for job 3...
Sending event for job 5...
Sending event for job 7...
Sending event for job 9...
Sending event for job 11...
Sending event for job 13...
Sending event for job 14...
Sending event for job 15...
Sending event for job 16...
Sending event for job 17...
Sending event for job 18...
Sending event for job 19...
Sending event for job 20...
Sending event for job 21...
Sending event for job 22...
Sending event for job 23...
Sending event for job 24...
Sending event for job 25...
Sending event for job 26...
Sending event for job 27...
Sending event for job 28...
Sending event for job 29...
Sending event for job 30...
Sending event for job 31...
Sending event for job 32...
Sending event for job 33...
Sending event for job 34...
Sending event for job 35...
Sending event for job 36...
Sending event for job 37...
Sending event for job 38...
Sending event for job 39...
Sending event for job 40...
Sending event for job 41...
Sending event for job 42...
Sending event for job 43...
Sending event for job 44...
Sending event for job 45...
Sending event for job 46...
Sending event for job 47...
Sending event for job 48...
Sending event for job 49...
Sending event for job 50...
Sending event for job 51...
Sending event for job 52...
Sending event for job 53...
Sending event for job 54...
Sending event for job 55...
Sending event for job 56...
Sending event for job 57...
Sending event for job 58...
Sending event for job 59...
Sending event for job 60...
Sending event for job 61...
Sending event for job 62...
Sending event for job 63...
Sending event for job 64...
Sending event for job 65...
Sending event for job 66...
Sending event for job 67...
Sending event for job 68...
Sending event for job 69...
Sending event for job 70...
Sending event for job 71...
Sending event for job 72...
Sending event for job 73...
Sending event for job 74...
Sending event for job 75...
Sending event for job 76...
Sending event for job 77...
Sending event for job 78...
Sending event for job 79...
Sending event for job 80...
Sending event for job 81...
Sending event for job 82...
Sending event for job 83...
Sending event for job 84...
Sending event for job 85...
Sending event for job 86...
Sending event for job 87...
Sending event for job 88...
Sending event for job 89...
Sending event for job 90...
Sending event for job 91...
Sending event for job 92...
Sending event for job 93...
Sending event for job 94...
Sending event for job 95...
Sending event for job 96...
Sending event for job 97...
Sending event for job 98...
Sending event for job 99...
Sending event for job 100...
Sending event for job 101...
Sending event for job 102...
Sending event for job 103...
Sending event for job 104...
Sending event for job 105...
Sending event for job 106...
Sending event for job 107...
Sending event for job 108...
Sending event for job 109...
Sending event for job 110...
Sending event for job 111...
Sending event for job 112...
Sending event for job 113...
Sending event for job 114...
Sending event for job 115...
Sending event for job 116...
Sending event for job 117...
Sending event for job 118...
Sending event for job 119...
Sending event for job 120...
Sending event for job 121...
Sending event for job 122...
Sending event for job 123...
Sending event for job 124...
Sending event for job 125...
Sending event for job 126...
Sending event for job 127...
Sending event for job 128...
Sending event for job 129...
Sending event for job 130...
Sending event for job 131...
Sending event for job 132...
Sending event for job 133...
Sending event for job 134...
Sending event for job 135...
Sending event for job 136...
Sending event for job 137...
Sending event for job 138...
Sending event for job 139...
Sending event for job 140...
Sending event for job 141...
Sending event for job 142...
Sending event for job 143...
Sending event for job 144...
Sending event for job 145...
Sending event for job 146...
Sending event for job 147...
Sending event for job 148...
Sending event for job 149...
Sending event for job 150...
Sending event for job 151...
Sending event for job 152...
Sending event for job 153...
Sending event for job 154...
Sending event for job 155...
Sending event for job 156...
Sending event for job 157...
Sending event for job 158...
Sending event for job 159...
Sending event for job 160...
Sending event for job 161...
Sending event for job 162...
Sending event for job 163...
Sending event for job 164...
Sending event for job 165...
Sending event for job 166...
Sending event for job 167...
Sending event for job 168...
Sending event for job 169...
Sending event for job 170...
Sending event for job 171...
Sending event for job 172...
Sending event for job 173...
Sending event for job 174...
Sending event for job 175...
Sending event for job 176...
Sending event for job 177...
Sending event for job 178...
Sending event for job 179...
Sending event for job 180...
Sending event for job 181...
Sending event for job 182...
Sending event for job 183...
Sending event for job 184...
Sending event for job 185...
Sending event for job 186...
Sending event for job 187...
Sending event for job 188...
Sending event for job 189...
Sending event for job 190...
Sending event for job 191...
Sending event for job 192...
Sending event for job 193...
Sending event for job 194...
Sending event for job 195...
Sending event for job 196...
Sending event for job 197...
Sending event for job 198...
Sending event for job 199...
Sending event for job 200...
Sending event for job 201...
Sending event for job 202...
Sending event for job 203...
Sending event for job 204...
Sending event for job 205...
Sending event for job 206...
Sending event for job 207...
Sending event for job 208...
Sending event for job 209...
Sending event for job 210...
Sending event for job 211...
Sending event for job 212...
Sending event for job 213...
Sending event for job 214...
Sending event for job 215...
Sending event for job 216...
Sending event for job 217...
Sending event for job 218...
Sending event for job 219...
Sending event for job 220...
Sending event for job 221...
Sending event for job 222...
Sending event for job 223...
Sending event for job 224...
Sending event for job 225...
Sending event for job 226...
Sending event for job 227...
Sending event for job 228...
Sending event for job 229...
Sending event for job 230...
Sending event for job 231...
Sending event for job 232...
Sending event for job 233...
Sending event for job 234...
Sending event for job 235...
Sending event for job 236...
Sending event for job 237...
Sending event for job 238...
Sending event for job 239...
Sending event for job 240...
Sending event for job 241...
Sending event for job 242...
Sending event for job 243...
Sending event for job 244...
Sending event for job 245...
Sending event for job 246...
Sending event for job 247...
Sending event for job 248...
Sending event for job 249...
Sending event for job 250...
Sending event for job 251...
Sending event for job 252...
Sending event for job 253...
Sending event for job 254...
Sending event for job 255...
Sending event for job 256...
Sending event for job 257...
Sending event for job 258...
Sending event for job 259...
Sending event for job 260...
Sending event for job 261...
Sending event for job 262...
Sending event for job 263...
Sending event for job 264...
Sending event for job 265...
Sending event for job 266...
Sending event for job 267...
Sending event for job 268...
Sending event for job 269...
Sending event for job 270...
Sending event for job 271...
Sending event for job 272...
Sending event for job 273...
Sending event for job 274...
Sending event for job 275...
Sending event for job 276...
Sending event for job 277...
Sending event for job 278...
Sending event for job 279...
Sending event for job 280...
Sending event for job 281...
Sending event for job 282...
Sending event for job 283...
Sending event for job 284...
Sending event for job 285...
Sending event for job 286...
Sending event for job 287...
Sending event for job 288...
Sending event for job 289...
Sending event for job 290...
Sending event for job 291...
Sending event for job 292...
Sending event for job 293...
Sending event for job 294...
Sending event for job 295...
Sending event for job 296...
Sending event for job 297...
Sending event for job 298...
Sending event for job 299...
Sending event for job 300...
Sending event for job 301...
Sending event for job 302...
Sending event for job 303...
Sending event for job 304...
Sending event for job 305...
Sending event for job 306...
Sending event for job 307...
Sending event for job 308...
Sending event for job 309...
Sending event for job 310...
Sending event for job 311...
Sending event for job 312...
Sending event for job 313...
Sending event for job 314...
Sending event for job 315...
Sending event for job 316...
Sending event for job 317...
Sending event for job 318...
Sending event for job 319...
Sending event for job 320...
Sending event for job 321...
Sending event for job 322...
Sending event for job 323...
Sending event for job 324...
Sending event for job 325...
Sending event for job 326...
Sending event for job 327...
Sending event for job 328...
Sending event for job 329...
Sending event for job 330...
Sending event for job 331...
Sending event for job 332...
Sending event for job 333...
Sending event for job 334...
Sending event for job 335...
Sending event for job 336...
Sending event for job 337...
Sending event for job 338...
Sending event for job 339...
Sending event for job 340...
Sending event for job 341...
Sending event for job 342...
Sending event for job 343...
Sending event for job 344...
Sending event for job 345...
Sending event for job 346...
Sending event for job 347...
Sending event for job 348...
Sending event for job 349...
Sending event for job 350...
Sending event for job 351...
Sending event for job 352...
Sending event for job 353...
Sending event for job 354...
Sending event for job 355...
Sending event for job 356...
Sending event for job 357...
Sending event for job 358...
Sending event for job 359...
Sending event for job 360...
Sending event for job 361...
Sending event for job 362...
Sending event for job 363...
Sending event for job 364...
Sending event for job 365...
Sending event for job 366...
Sending event for job 367...
Sending event for job 368...
Sending event for job 369...
Sending event for job 370...
Sending event for job 371...
Sending event for job 372...
Sending event for job 373...
Sending event for job 374...
Sending event for job 375...
Sending event for job 376...
Sending event for job 377...
Sending event for job 378...
Sending event for job 379...
Sending event for job 380...
Sending event for job 381...
Sending event for job 382...
Sending event for job 383...
Sending event for job 384...
Sending event for job 385...
Sending event for job 386...
Sending event for job 387...
Sending event for job 388...
Sending event for job 389...
Sending event for job 390...
Sending event for job 391...
Sending event for job 392...
Sending event for job 393...
Sending event for job 394...
Sending event for job 395...
Sending event for job 396...
Sending event for job 397...
Sending event for job 398...
Sending event for job 399...
Sending event for job 400...
Sending event for job 401...
Sending event for job 402...
Sending event for job 403...
Sending event for job 404...
Sending event for job 405...
Sending event for job 406...
Sending event for job 407...
Sending event for job 408...
Sending event for job 409...
Sending event for job 410...
Sending event for job 411...
Sending event for job 412...
Sending event for job 413...
Sending event for job 414...
Sending event for job 415...
Sending event for job 416...
Sending event for job 417...
Sending event for job 418...
Sending event for job 419...
Sending event for job 420...
Sending event for job 421...
Sending event for job 422...
Sending event for job 423...
Sending event for job 424...
Sending event for job 425...
Sending event for job 426...
Sending event for job 427...
Sending event for job 428...
Sending event for job 429...
Sending event for job 430...
Sending event for job 431...
Sending event for job 432...
Sending event for job 433...
Sending event for job 434...
Sending event for job 435...
Sending event for job 436...
Sending event for job 437...
Sending event for job 438...
Sending event for job 439...
Sending event for job 440...
Sending event for job 441...
Sending event for job 442...
Sending event for job 443...
Sending event for job 444...
Sending event for job 445...
Sending event for job 446...
Sending event for job 447...
Sending event for job 448...
Sending event for job 449...
Sending event for job 450...
Sending event for job 451...
Sending event for job 452...
Sending event for job 453...
Sending event for job 454...
Sending event for job 455...
Sending event for job 456...
Sending event for job 457...
Sending event for job 458...
Sending event for job 459...
Sending event for job 460...
Sending event for job 461...
Sending event for job 462...
Sending event for job 463...
Sending event for job 464...
Sending event for job 465...
Sending event for job 466...
Sending event for job 467...
Sending event for job 468...
Sending event for job 469...
Sending event for job 470...
Sending event for job 471...
Sending event for job 472...
Sending event for job 473...
Sending event for job 474...
Sending event for job 475...
Sending event for job 476...
Sending event for job 477...
Sending event for job 478...
Sending event for job 479...
Sending event for job 480...
Sending event for job 481...
Sending event for job 482...
Sending event for job 483...
Sending event for job 484...
Sending event for job 485...
Sending event for job 486...
Sending event for job 487...
Sending event for job 488...
Sending event for job 489...
Sending event for job 490...
Sending event for job 491...
Sending event for job 492...
Sending event for job 493...
Sending event for job 494...
Sending event for job 495...
Sending event for job 496...
Sending event for job 497...
Sending event for job 498...
Sending event for job 499...
Sending event for job 500...
Sending event for job 501...
Sending event for job 502...
Sending event for job 503...
Sending event for job 504...
Sending event for job 505...
Sending event for job 506...
Sending event for job 507...
Sending event for job 508...
Sending event for job 509...
Sending event for job 510...
Sending event for job 511...
Sending event for job 512...
Sending event for job 513...
Sending event for job 514...
Sending event for job 515...
Sending event for job 516...
Sending event for job 517...
Sending event for job 518...
Sending event for job 519...
Sending event for job 520...
Sending event for job 521...
Sending event for job 522...
Sending event for job 523...
Sending event for job 524...
Sending event for job 525...
Sending event for job 526...
Sending event for job 527...
Sending event for job 528...
Sending event for job 529...
Sending event for job 530...
Sending event for job 531...
Sending event for job 532...
Sending event for job 533...
Sending event for job 534...
Sending event for job 535...
Sending event for job 536...
Sending event for job 537...
Sending event for job 538...
Sending event for job 539...
Sending event for job 540...
Sending event for job 541...
Sending event for job 542...
Sending event for job 543...
Sending event for job 544...
Sending event for job 545...
Sending event for job 546...
Sending event for job 547...
Sending event for job 548...
Sending event for job 549...
Sending event for job 550...
Sending event for job 551...
Sending event for job 552...
Sending event for job 553...
Sending event for job 554...
Sending event for job 555...
Sending event for job 556...
Sending event for job 557...
Sending event for job 558...
Sending event for job 559...
Sending event for job 560...
Sending event for job 561...
Sending event for job 562...
Sending event for job 563...
Sending event for job 564...
Sending event for job 565...
Sending event for job 566...
Sending event for job 567...
Sending event for job 568...
Sending event for job 569...
Sending event for job 570...
Sending event for job 571...
Sending event for job 572...
Sending event for job 573...
Sending event for job 574...
Sending event for job 575...
Sending event for job 576...
Sending event for job 577...
Sending event for job 578...
Sending event for job 579...
Sending event for job 580...
Sending event for job 581...
Sending event for job 582...
Sending event for job 583...
Sending event for job 584...
Sending event for job 585...
Sending event for job 586...
Sending event for job 587...
Sending event for job 588...
Sending event for job 589...
Sending event for job 590...
Sending event for job 591...
Sending event for job 592...
Sending event for job 593...
Sending event for job 594...
Sending event for job 595...
Sending event for job 596...
Sending event for job 597...
Sending event for job 598...
Sending event for job 599...
Sending event for job 600...
Sending event for job 601...
Sending event for job 602...
Sending event for job 603...
Sending event for job 604...
Sending event for job 605...
Sending event for job 606...
Sending event for job 607...
Sending event for job 608...
Sending event for job 609...
Sending event for job 610...
Sending event for job 611...
Sending event for job 612...
Sending event for job 613...
Sending event for job 614...
Sending event for job 615...
Sending event for job 616...
Sending event for job 617...
Sending event for job 618...
Sending event for job 619...
Sending event for job 620...
Sending event for job 621...
Sending event for job 622...
Sending event for job 623...
Sending event for job 624...
Sending event for job 625...
Sending event for job 626...
Sending event for job 627...
Sending event for job 628...
Sending event for job 629...
Sending event for job 630...
Sending event for job 631...
Sending event for job 632...
Sending event for job 633...
Sending event for job 634...
Sending event for job 635...
Sending event for job 636...
Sending event for job 637...
Sending event for job 638...
Sending event for job 639...
Sending event for job 640...
Sending event for job 641...
Sending event for job 642...
Sending event for job 643...
Sending event for job 644...
Sending event for job 645...
Sending event for job 646...
Sending event for job 647...
Sending event for job 648...
Sending event for job 649...
Sending event for job 650...
Sending event for job 651...
Sending event for job 652...
Sending event for job 653...
Sending event for job 654...
Sending event for job 655...
Sending event for job 656...
Sending event for job 657...
Sending event for job 658...
Sending event for job 659...
Sending event for job 660...
Sending event for job 661...
Sending event for job 662...
Sending event for job 663...
Sending event for job 664...
Sending event for job 665...
Sending event for job 666...
Sending event for job 667...
Sending event for job 668...
Sending event for job 669...
Sending event for job 670...
Sending event for job 671...
Sending event for job 672...
Sending event for job 673...
Sending event for job 674...
Sending event for job 675...
Sending event for job 676...
Sending event for job 677...
Sending event for job 678...
Sending event for job 679...
Sending event for job 680...
Sending event for job 681...
Sending event for job 682...
Sending event for job 683...
Sending event for job 684...
Sending event for job 685...
Sending event for job 686...
Sending event for job 687...
Sending event for job 688...
Sending event for job 689...
Sending event for job 690...
Sending event for job 691...
Sending event for job 692...
Sending event for job 693...
Sending event for job 694...
Sending event for job 695...
Sending event for job 696...
Sending event for job 698...
Sending event for job 699...
Sending event for job 700...
Sending event for job 701...
Sending event for job 702...
Sending event for job 703...
Sending event for job 705...
Sending event for job 706...
Sending event for job 707...
Sending event for job 708...
Sending event for job 710...
Sending event for job 711...
Sending event for job 712...
Sending event for job 713...
Sending event for job 714...
Sending event for job 715...
Sending event for job 716...
Sending event for job 717...
Sending event for job 718...
Sending event for job 719...
Sending event for job 720...
Sending event for job 721...
Sending event for job 722...
Sending event for job 723...
Sending event for job 724...
Sending event for job 725...
Sending event for job 726...
Sending event for job 727...
Sending event for job 728...
Sending event for job 729...
Sending event for job 730...
Sending event for job 731...
Sending event for job 732...
Sending event for job 733...
Sending event for job 734...
Sending event for job 735...
Sending event for job 736...
Sending event for job 737...
Sending event for job 738...
Sending event for job 739...
Sending event for job 740...
Sending event for job 741...
Sending event for job 742...
Sending event for job 743...
Sending event for job 744...
Sending event for job 745...
Sending event for job 747...
Sending event for job 749...
Sending event for job 751...
Sending event for job 753...
Sending event for job 755...
Sending event for job 757...
Sending event for job 760...
Sending event for job 761...
Sending event for job 763...
Sending event for job 765...
Sending event for job 767...
Sending event for job 769...
Sending event for job 771...
Sending event for job 773...
Sending event for job 775...
Sending event for job 777...
Sending event for job 779...
Sending event for job 781...
Sending event for job 783...
Sending event for job 785...
Sending event for job 787...
Sending event for job 789...
Sending event for job 791...
Sending event for job 793...
Sending event for job 795...
Sending event for job 797...
Sending event for job 799...
Sending event for job 801...
Sending event for job 803...
Sending event for job 805...
Sending event for job 807...
Sending event for job 809...
Sending event for job 811...
Sending event for job 813...
Sending event for job 815...
Sending event for job 817...
Sending event for job 819...
Sending event for job 821...
Sending event for job 823...
Sending event for job 825...
Sending event for job 828...
Sending event for job 830...
Sending event for job 832...
Sending event for job 834...
Sending event for job 836...
Sending event for job 838...
Sending event for job 840...
Sending event for job 842...
Sending event for job 844...
Sending event for job 846...
Sending event for job 848...
Sending event for job 850...
Sending event for job 853...
Sending event for job 854...
Sending event for job 856...
Sending event for job 858...
Sending event for job 860...
Sending event for job 862...
Sending event for job 864...
Sending event for job 865...
Sending event for job 867...
Sending event for job 869...
Sending event for job 871...
Sending event for job 873...
Sending event for job 875...
Sending event for job 877...
Sending event for job 879...
Sending event for job 881...
Sending event for job 883...
Sending event for job 885...
Sending event for job 887...
Sending event for job 889...
Sending event for job 891...
Sending event for job 893...
Sending event for job 896...
Sending event for job 897...
Sending event for job 900...
Sending event for job 901...
Sending event for job 903...
Sending event for job 905...
Sending event for job 907...
Sending event for job 909...
Sending event for job 911...
Sending event for job 913...
Sending event for job 915...
Sending event for job 918...
Sending event for job 919...
Sending event for job 921...
Sending event for job 923...
Sending event for job 925...
Sending event for job 927...
Sending event for job 929...
Sending event for job 930...
Sending event for job 931...
Sending event for job 932...
Sending event for job 933...
Sending event for job 934...
Sending event for job 935...
Sending event for job 936...
Sending event for job 937...
Sending event for job 938...
Sending event for job 939...
Sending event for job 940...
Sending event for job 941...
Sending event for job 942...
Sending event for job 943...
Sending event for job 944...
Sending event for job 945...
Sending event for job 946...
Sending event for job 947...
Sending event for job 948...
Sending event for job 949...
Sending event for job 950...
Sending event for job 951...
Sending event for job 952...
Sending event for job 953...
Sending event for job 954...
Sending event for job 955...
Sending event for job 956...
Sending event for job 957...
Sending event for job 958...
Sending event for job 959...
Sending event for job 960...
Sending event for job 961...
Sending event for job 962...
Sending event for job 963...
Sending event for job 964...
Sending event for job 965...
Sending event for job 966...
Sending event for job 967...
Sending event for job 968...
Sending event for job 969...
Sending event for job 970...
Sending event for job 971...
Sending event for job 972...
Sending event for job 973...
Sending event for job 974...
Sending event for job 975...
Sending event for job 976...
Sending event for job 977...
Sending event for job 978...
Sending event for job 979...
Sending event for job 980...
Sending event for job 981...
Sending event for job 982...
Sending event for job 983...
Sending event for job 984...
Sending event for job 985...
Sending event for job 986...
Sending event for job 987...
Sending event for job 988...
Sending event for job 989...
Sending event for job 990...
Sending event for job 991...
Sending event for job 992...
Sending event for job 993...
Sending event for job 994...
Sending event for job 995...
Sending event for job 996...
Sending event for job 997...
Sending event for job 998...
Sending event for job 999...
Sending event for job 1000...
Sending event for job 1001...
Sending event for job 1002...
Sending event for job 1003...
Sending event for job 1004...
Sending event for job 1005...
Sending event for job 1006...
Sending event for job 1007...
Sending event for job 1008...
Sending event for job 1009...
Sending event for job 1010...
Sending event for job 1011...
Sending event for job 1012...
Sending event for job 1013...
Sending event for job 1014...
Sending event for job 1015...
Sending event for job 1016...
Sending event for job 1017...
Sending event for job 1018...
Sending event for job 1019...
Sending event for job 1020...
Sending event for job 1021...
Sending event for job 1022...
Sending event for job 1023...
Sending event for job 1024...
Sending event for job 1025...
Sending event for job 1026...
Sending event for job 1027...
Sending event for job 1028...
Sending event for job 1029...
Sending event for job 1030...
Sending event for job 1031...
Sending event for job 1032...
Sending event for job 1033...
Sending event for job 1034...
Sending event for job 1035...
Sending event for job 1036...
Sending event for job 1037...
Sending event for job 1038...
Sending event for job 1039...
Sending event for job 1040...
Sending event for job 1041...
Sending event for job 1042...
Sending event for job 1043...
Sending event for job 1044...
Sending event for job 1045...
Sending event for job 1046...
Sending event for job 1047...
Sending event for job 1048...
Sending event for job 1049...
Sending event for job 1050...
Sending event for job 1051...
Sending event for job 1052...
Sending event for job 1053...
Sending event for job 1054...
Sending event for job 1055...
Sending event for job 1056...
Sending event for job 1057...
Sending event for job 1058...
Sending event for job 1059...
Sending event for job 1060...
Sending event for job 1061...
Sending event for job 1062...
Sending event for job 1063...
Sending event for job 1064...
Sending event for job 1065...
Sending event for job 1066...
Sending event for job 1067...
Sending event for job 1068...
Sending event for job 1069...
Sending event for job 1070...
Sending event for job 1071...
Sending event for job 1072...
Sending event for job 1073...
Sending event for job 1074...
Sending event for job 1075...
Sending event for job 1076...
Sending event for job 1077...
Sending event for job 1078...
Sending event for job 1079...
Sending event for job 1080...
Sending event for job 1081...
Sending event for job 1082...
Sending event for job 1083...
Sending event for job 1084...
Sending event for job 1085...
Sending event for job 1086...
Sending event for job 1087...
Sending event for job 1088...
Sending event for job 1089...
Sending event for job 1090...
Sending event for job 1091...
Sending event for job 1092...
Sending event for job 1093...
Sending event for job 1094...
Sending event for job 1095...
Sending event for job 1096...
Sending event for job 1097...
Sending event for job 1098...
Sending event for job 1099...
Sending event for job 1100...
Sending event for job 1101...
Sending event for job 1102...
Sending event for job 1103...
Sending event for job 1104...
Sending event for job 1105...
Sending event for job 1106...
Sending event for job 1107...
Sending event for job 1108...
Sending event for job 1109...
Sending event for job 1110...
Sending event for job 1111...
Sending event for job 1112...
Sending event for job 1113...
Sending event for job 1114...
Sending event for job 1115...
Sending event for job 1116...
Sending event for job 1117...
Sending event for job 1118...
Sending event for job 1119...
Sending event for job 1120...
Sending event for job 1121...
Sending event for job 1122...
Sending event for job 1123...
Sending event for job 1124...
Sending event for job 1125...
Sending event for job 1126...
Sending event for job 1127...
Sending event for job 1128...
Sending event for job 1129...
Sending event for job 1130...
Sending event for job 1131...
Sending event for job 1132...
Sending event for job 1133...
Sending event for job 1134...
Sending event for job 1135...
Sending event for job 1136...
Sending event for job 1137...
Sending event for job 1144...
Sending event for job 1145...
Sending event for job 1146...
Sending event for job 1147...
Sending event for job 1148...
Sending event for job 1149...
Sending event for job 1150...
Sending event for job 1151...
Sending event for job 1152...
Sending event for job 1153...
Sending event for job 1154...
Sending event for job 1155...
Sending event for job 1156...
Sending event for job 1157...
Sending event for job 1158...
Sending event for job 1159...
Sending event for job 1160...
Sending event for job 1161...
Sending event for job 1162...
Sending event for job 1163...
Sending event for job 1164...
Sending event for job 1165...
Sending event for job 1166...
Sending event for job 1167...
Sending event for job 1168...
Sending event for job 1169...
Sending event for job 1170...
Sending event for job 1171...
Sending event for job 1172...
Sending event for job 1173...
Sending event for job 1174...
Sending event for job 1175...
Sending event for job 1176...
Sending event for job 1177...
Sending event for job 1178...
Sending event for job 1179...
Sending event for job 1180...
Sending event for job 1181...
Sending event for job 1182...
Sending event for job 1183...
Sending event for job 1184...
Sending event for job 1185...
Sending event for job 1186...
Sending event for job 1187...
Sending event for job 1188...
Sending event for job 1189...
Sending event for job 1190...
Sending event for job 1191...
Sending event for job 1192...
Sending event for job 1193...
Sending event for job 1194...
Sending event for job 1195...
Sending event for job 1196...
Sending event for job 1197...
Sending event for job 1198...
Sending event for job 1199...
Sending event for job 1200...
Sending event for job 1201...
Sending event for job 1202...
Sending event for job 1203...
Sending event for job 1204...
Sending event for job 1205...
Sending event for job 1206...
Sending event for job 1207...
Sending event for job 1208...
Sending event for job 1209...
Sending event for job 1210...
Sending event for job 1211...
Sending event for job 1212...
Sending event for job 1213...
Sending event for job 1214...
Sending event for job 1215...
Sending event for job 1216...
Sending event for job 1217...
Sending event for job 1218...
Sending event for job 1219...
Sending event for job 1220...
Sending event for job 1221...
Sending event for job 1222...
Sending event for job 1223...
Sending event for job 1224...
Sending event for job 1225...
Sending event for job 1226...
Sending event for job 1227...
Sending event for job 1228...
Sending event for job 1229...
Sending event for job 1230...
Sending event for job 1231...
Sending event for job 1232...
Sending event for job 1233...
Sending event for job 1234...
Sending event for job 1235...
Sending event for job 1236...
Sending event for job 1237...
Sending event for job 1238...
Sending event for job 1239...
Sending event for job 1240...
Sending event for job 1241...
Sending event for job 1242...
Sending event for job 1243...
Sending event for job 1244...
Sending event for job 1245...
Sending event for job 1246...
Sending event for job 1247...
Sending event for job 1248...
Sending event for job 1249...
Sending event for job 1250...
Sending event for job 1251...
Sending event for job 1252...
Sending event for job 1253...
Sending event for job 1254...
Sending event for job 1255...
Sending event for job 1256...
Sending event for job 1257...
Sending event for job 1258...
Sending event for job 1259...
Sending event for job 1260...
Sending event for job 1261...
Sending event for job 1262...
Sending event for job 1263...
Sending event for job 1264...
Sending event for job 1265...
Sending event for job 1266...
Sending event for job 1267...
Sending event for job 1268...
Sending event for job 1269...
Sending event for job 1270...
Sending event for job 1271...
Sending event for job 1272...
Sending event for job 1273...
Sending event for job 1274...
Sending event for job 1275...
Sending event for job 1276...
Sending event for job 1277...
Sending event for job 1278...
Sending event for job 1279...
Sending event for job 1280...
Sending event for job 1281...
Sending event for job 1282...
Sending event for job 1283...
Sending event for job 1284...
Sending event for job 1285...
Sending event for job 1286...
Sending event for job 1287...
Sending event for job 1288...
Sending event for job 1289...
Sending event for job 1290...
Sending event for job 1291...
Sending event for job 1292...
Sending event for job 1293...
Sending event for job 1294...
Sending event for job 1295...
Sending event for job 1296...
Sending event for job 1297...
Sending event for job 1298...
Sending event for job 1299...
Sending event for job 1300...
Sending event for job 1301...
Sending event for job 1302...
Sending event for job 1303...
Sending event for job 1304...
Sending event for job 1305...
Sending event for job 1306...
Sending event for job 1307...
Sending event for job 1308...
Sending event for job 1309...
Sending event for job 1310...
Sending event for job 1311...
Sending event for job 1312...
Sending event for job 1313...
Sending event for job 1314...
Sending event for job 1315...
Sending event for job 1316...
Sending event for job 1317...
Sending event for job 1318...
Sending event for job 1319...
Sending event for job 1320...
Sending event for job 1321...
Sending event for job 1322...
Sending event for job 1323...
Sending event for job 1324...
Sending event for job 1325...
Sending event for job 1326...
Sending event for job 1327...
Sending event for job 1328...
Sending event for job 1329...
Sending event for job 1330...
Sending event for job 1331...
Sending event for job 1332...
Sending event for job 1333...
Sending event for job 1334...
Sending event for job 1335...
Sending event for job 1336...
Sending event for job 1337...
Sending event for job 1338...
Sending event for job 1339...
Sending event for job 1340...
Sending event for job 1341...
Sending event for job 1342...
Sending event for job 1343...
Sending event for job 1344...
Sending event for job 1345...
Sending event for job 1346...
Sending event for job 1347...
Sending event for job 1348...
Sending event for job 1349...
Sending event for job 1350...
Sending event for job 1351...
Sending event for job 1352...
Sending event for job 1353...
Sending event for job 1354...
Sending event for job 1355...
Sending event for job 1356...
Sending event for job 1357...
Sending event for job 1358...
Sending event for job 1359...
Sending event for job 1360...
Sending event for job 1361...
Sending event for job 1362...
Sending event for job 1363...
Sending event for job 1364...
Sending event for job 1365...
Sending event for job 1366...
Sending event for job 1367...
Sending event for job 1368...
Sending event for job 1369...
Sending event for job 1370...
Sending event for job 1371...
Sending event for job 1372...
Sending event for job 1373...
Sending event for job 1374...
Sending event for job 1375...
Sending event for job 1376...
Sending event for job 1377...
Sending event for job 1378...
Sending event for job 1379...
Sending event for job 1385...
Sending event for job 1386...
Sending event for job 1387...
Sending event for job 1388...
Sending event for job 1389...
Sending event for job 1390...
Sending event for job 1391...
Sending event for job 1392...
Sending event for job 1393...
Sending event for job 1394...
Sending event for job 1395...
Sending event for job 1396...
Sending event for job 1397...
Sending event for job 1398...
Sending event for job 1399...
Sending event for job 1400...
Sending event for job 1401...
Sending event for job 1402...
Sending event for job 1403...
Sending event for job 1404...
Sending event for job 1405...
Sending event for job 1406...
Sending event for job 1407...
Sending event for job 1408...
Sending event for job 1409...
Sending event for job 1410...
Sending event for job 1411...
Sending event for job 1412...
Sending event for job 1413...
Sending event for job 1414...
Sending event for job 1415...
Sending event for job 1416...
Sending event for job 1417...
Sending event for job 1418...
Sending event for job 1419...
Sending event for job 1420...
Sending event for job 1421...
Sending event for job 1422...
Sending event for job 1423...
Sending event for job 1424...
Sending event for job 1425...
Sending event for job 1426...
Sending event for job 1427...
Sending event for job 1428...
Sending event for job 1429...
Sending event for job 1430...
Sending event for job 1431...
Sending event for job 1432...
Sending event for job 1433...
Sending event for job 1434...
Sending event for job 1435...
Sending event for job 1436...
Sending event for job 1437...
Sending event for job 1438...
Sending event for job 1439...
Sending event for job 1440...
Sending event for job 1441...
Sending event for job 1442...
Sending event for job 1443...
Sending event for job 1444...
Sending event for job 1445...
Sending event for job 1446...
Sending event for job 1447...
Sending event for job 1448...
Sending event for job 1454...
Sending event for job 1455...
Sending event for job 1456...
Sending event for job 1457...
Sending event for job 1458...
Sending event for job 1459...
Sending event for job 1460...
Sending event for job 1461...
Sending event for job 1462...
Sending event for job 1463...
Sending event for job 1464...
Sending event for job 1465...
Sending event for job 1466...
Sending event for job 1467...
Sending event for job 1468...
Sending event for job 1469...
Sending event for job 1470...
Sending event for job 1471...
Sending event for job 1472...
Sending event for job 1473...
Sending event for job 1474...
Sending event for job 1475...
Sending event for job 1476...
Sending event for job 1477...
Sending event for job 1478...
Sending event for job 1479...
Sending event for job 1480...
Sending event for job 1481...
Sending event for job 1482...
Sending event for job 1483...
Sending event for job 1484...
Sending event for job 1485...
Sending event for job 1486...
Sending event for job 1487...
Sending event for job 1488...
Sending event for job 1489...
Sending event for job 1490...
Sending event for job 1491...
Sending event for job 1492...
Sending event for job 1493...
Sending event for job 1494...
Sending event for job 1495...
Sending event for job 1496...
Sending event for job 1497...
Sending event for job 1498...
Sending event for job 1499...
Sending event for job 1500...
Sending event for job 1501...
Sending event for job 1502...
Sending event for job 1503...
Sending event for job 1504...
Sending event for job 1505...
Sending event for job 1506...
Sending event for job 1507...
Sending event for job 1508...
Sending event for job 1509...
Sending event for job 1510...
Sending event for job 1511...
Sending event for job 1512...
Sending event for job 1513...
Sending event for job 1514...
Sending event for job 1520...
Sending event for job 1521...
Sending event for job 1522...
Sending event for job 1523...
Sending event for job 1524...
Sending event for job 1525...
Sending event for job 1526...
Sending event for job 1527...
Sending event for job 1528...
Sending event for job 1529...
Sending event for job 1530...
Sending event for job 1531...
Sending event for job 1532...
Sending event for job 1533...
Sending event for job 1534...
Sending event for job 1535...
Sending event for job 1536...
Sending event for job 1537...
Sending event for job 1538...
Sending event for job 1539...
Sending event for job 1540...
Sending event for job 1541...
Sending event for job 1542...
Sending event for job 1543...
Sending event for job 1544...
Sending event for job 1545...
Sending event for job 1546...
Sending event for job 1547...
Sending event for job 1548...
Sending event for job 1549...
Sending event for job 1550...
Sending event for job 1551...
Sending event for job 1552...
Sending event for job 1553...
Sending event for job 1554...
Sending event for job 1555...
Sending event for job 1556...
Sending event for job 1557...
Sending event for job 1558...
Sending event for job 1559...
Sending event for job 1560...
Sending event for job 1561...
Sending event for job 1562...
Sending event for job 1563...
Sending event for job 1564...
Sending event for job 1565...
Sending event for job 1566...
Sending event for job 1567...
Sending event for job 1574...
Sending event for job 1575...
Sending event for job 1576...
Sending event for job 1577...
Sending event for job 1578...
Sending event for job 1579...
Sending event for job 1580...
Sending event for job 1581...
Sending event for job 1582...
Sending event for job 1583...
Sending event for job 1584...
Sending event for job 1585...
Sending event for job 1586...
Sending event for job 1587...
Sending event for job 1588...
Sending event for job 1589...
Sending event for job 1590...
Sending event for job 1591...
Sending event for job 1592...
Sending event for job 1593...
Sending event for job 1594...
Sending event for job 1595...
Sending event for job 1596...
Sending event for job 1597...
Sending event for job 1598...
Sending event for job 1599...
Sending event for job 1600...
Sending event for job 1601...
Sending event for job 1602...
Sending event for job 1603...
Sending event for job 1604...
Sending event for job 1605...
Sending event for job 1606...
Sending event for job 1607...
Sending event for job 1608...
Sending event for job 1609...
Sending event for job 1610...
Sending event for job 1611...
Sending event for job 1612...
Sending event for job 1613...
Sending event for job 1614...
Sending event for job 1615...
Sending event for job 1616...
Sending event for job 1617...
Sending event for job 1618...
Sending event for job 1619...
Sending event for job 1620...
Sending event for job 1621...
Sending event for job 1622...
Sending event for job 1623...
Sending event for job 1624...
Sending event for job 1625...
Sending event for job 1626...
Sending event for job 1627...
Sending event for job 1628...
Sending event for job 1629...
Sending event for job 1630...
Sending event for job 1631...
Sending event for job 1632...
Sending event for job 1639...
Sending event for job 1640...
Sending event for job 1641...
Sending event for job 1642...
Sending event for job 1643...
Sending event for job 1644...
Sending event for job 1645...
Sending event for job 1646...
Sending event for job 1647...
Sending event for job 1648...
Sending event for job 1649...
Sending event for job 1650...
Sending event for job 1651...
Sending event for job 1652...
Sending event for job 1653...
Sending event for job 1654...
Sending event for job 1655...
Sending event for job 1656...
Sending event for job 1657...
Sending event for job 1658...
Sending event for job 1659...
Sending event for job 1660...
Sending event for job 1661...
Sending event for job 1662...
Sending event for job 1663...
Sending event for job 1664...
Sending event for job 1665...
Sending event for job 1666...
Sending event for job 1667...
Sending event for job 1668...
Sending event for job 1669...
Sending event for job 1670...
Sending event for job 1671...
Sending event for job 1672...
Sending event for job 1673...
Sending event for job 1674...
Sending event for job 1675...
Sending event for job 1676...
Sending event for job 1677...
Sending event for job 1678...
Sending event for job 1685...
Sending event for job 1686...
Sending event for job 1687...
Sending event for job 1688...
Sending event for job 1689...
Sending event for job 1690...
Sending event for job 1691...
Sending event for job 1692...
Sending event for job 1693...
Sending event for job 1694...
Sending event for job 1695...
Sending event for job 1696...
Sending event for job 1697...
Sending event for job 1698...
Sending event for job 1699...
Sending event for job 1700...
Sending event for job 1701...
Sending event for job 1702...
Sending event for job 1703...
Sending event for job 1704...
Sending event for job 1705...
Sending event for job 1706...
Sending event for job 1707...
Sending event for job 1708...
Sending event for job 1709...
Sending event for job 1710...
Sending event for job 1711...
Sending event for job 1712...
Sending event for job 1713...
Sending event for job 1714...
Sending event for job 1715...
Sending event for job 1716...
Sending event for job 1717...
Sending event for job 1718...
Sending event for job 1719...
Sending event for job 1720...
Sending event for job 1721...
Sending event for job 1722...
Sending event for job 1723...
Sending event for job 1724...
Sending event for job 1725...
Sending event for job 1732...
Sending event for job 1733...
Sending event for job 1734...
Sending event for job 1735...
Sending event for job 1736...
Sending event for job 1737...
Sending event for job 1738...
Sending event for job 1739...
Sending event for job 1740...
Sending event for job 1741...
Sending event for job 1742...
Sending event for job 1743...
Sending event for job 1744...
Sending event for job 1745...
Sending event for job 1746...
Sending event for job 1747...
Sending event for job 1748...
Sending event for job 1749...
Sending event for job 1750...
Sending event for job 1751...
Sending event for job 1752...
Sending event for job 1753...
Sending event for job 1754...
Sending event for job 1755...
Sending event for job 1756...
Sending event for job 1757...
Sending event for job 1758...
Sending event for job 1759...
Sending event for job 1760...
Sending event for job 1761...
Sending event for job 1762...
Sending event for job 1763...
Sending event for job 1380...
Sending event for job 1381...
Sending event for job 1382...
Sending event for job 1383...
Sending event for job 1384...
Sending event for job 1449...
Sending event for job 1450...
Sending event for job 1451...
Sending event for job 1452...
Sending event for job 1453...
Sending event for job 1515...
Sending event for job 1516...
Sending event for job 1517...
Sending event for job 1518...
Sending event for job 1519...
Sending event for job 1568...
Sending event for job 1569...
Sending event for job 1570...
Sending event for job 1571...
Sending event for job 1572...
Sending event for job 1573...
Sending event for job 1633...
Sending event for job 1634...
Sending event for job 1635...
Sending event for job 1636...
Sending event for job 1637...
Sending event for job 1638...
Sending event for job 1679...
Sending event for job 1680...
Sending event for job 1681...
Sending event for job 1682...
Sending event for job 1683...
Sending event for job 1684...
Sending event for job 1726...
Sending event for job 1727...
Sending event for job 1728...
Sending event for job 1729...
Sending event for job 1730...
Sending event for job 1731...
Sending event for job 1764...
Sending event for job 1765...
Sending event for job 1766...
Sending event for job 1767...
Sending event for job 1768...
Sending event for job 1769...

Querying the dataset

Querying a Honeycomb dataset in a particular way will produce a series of events (or a time series). The query builder is great tool for fast exploration from the Honeycomb UI. It is also possible to specify queries programatically.

The SQuaSH validate_drp.AM1 metric on a given dataset and filter can be obtained from this query:

In [6]:
query = {
    "breakdowns": [
        "dataset", "filter_name"
    ],
    "calculations": [
        {"column": "validate_drp.AM1", "op": "AVG"}
    ],
    "filters":[
      {"column": "dataset", "op": "=", "value": "validation_data_hsc"},
      {"column": "filter_name", "op": "=", "value": "HSC-R"}
    ],
    "filter_combination": "AND"
}

Markers

Markers are annotations over the time series plot. Markers are defined per dataset and can be created programatically via the UI, a CLI, or the Markers API.

In [7]:
headers = {'X-Honeycomb-Team': HONEY_API_KEY}

marker={"message": "Testing marker API"}

r = requests.post("https://api.honeycomb.io/1/markers/squash-demo", json=marker, headers=headers)
r.json()
Out[7]:
{'created_at': '2018-09-21T18:11:29Z',
 'updated_at': '2018-09-21T18:11:29Z',
 'start_time': 1537553489,
 'message': 'Testing marker API',
 'id': 'eiEzAM4KH6k'}

All the Markers in a dataset may be retrieved by:

In [8]:
r = requests.get("https://api.honeycomb.io/1/markers/squash-demo", headers=headers)
r.json()
Out[8]:
[{'created_at': '2018-09-06T22:57:40Z',
  'updated_at': '2018-09-06T22:57:40Z',
  'start_time': 1533460436,
  'message': 'Another markes',
  'type': 'Custom',
  'id': 'iGm7GHW6hHg'},
 {'created_at': '2018-09-06T22:55:56Z',
  'updated_at': '2018-09-06T22:55:56Z',
  'start_time': 1536274556,
  'message': 'Testing marker API',
  'id': 'GrD43u7xE5H'},
 {'created_at': '2018-09-10T17:27:19Z',
  'updated_at': '2018-09-10T17:27:19Z',
  'start_time': 1536600439,
  'message': 'Testing marker API',
  'id': 'hwNxeBnkbCH'},
 {'created_at': '2018-09-10T20:05:22Z',
  'updated_at': '2018-09-10T20:05:22Z',
  'start_time': 1536609922,
  'message': 'Testing marker API',
  'id': 'v1iwcV1obn'},
 {'created_at': '2018-09-10T20:37:53Z',
  'updated_at': '2018-09-10T20:37:53Z',
  'start_time': 1536611873,
  'message': 'Testing marker API',
  'id': 'ftBSdn81rGv'},
 {'created_at': '2018-09-10T20:47:34Z',
  'updated_at': '2018-09-10T20:47:34Z',
  'start_time': 1536612454,
  'message': 'Testing marker API',
  'id': 'vcpAZqn3hfV'},
 {'created_at': '2018-09-10T20:50:25Z',
  'updated_at': '2018-09-10T20:50:25Z',
  'start_time': 1536612625,
  'message': 'Testing marker API',
  'id': 'pbnaLumMHfH'},
 {'created_at': '2018-09-10T21:09:50Z',
  'updated_at': '2018-09-10T21:09:50Z',
  'start_time': 1536613790,
  'message': 'Testing marker API',
  'id': 'rRJqJqEb2H4'},
 {'created_at': '2018-09-21T17:42:09Z',
  'updated_at': '2018-09-21T17:42:09Z',
  'start_time': 1537551729,
  'message': 'Testing marker API',
  'id': '6gQBtgpktJH'},
 {'created_at': '2018-09-21T18:11:29Z',
  'updated_at': '2018-09-21T18:11:29Z',
  'start_time': 1537553489,
  'message': 'Testing marker API',
  'id': 'eiEzAM4KH6k'}]

Triggers

Given a time series and a threshold one can use the Triggers API for setting alerts when the values pass a threshold. Triggers are also defined per dataset. Honeycomb provides integration with Slack for alert notification.

Let’s set an alert for the validate_drp.AM1 metric. We’ll get the design specification for that metric from the SQuaSH API:

In [9]:
spec = requests.get(SQUASH_API_URL + "spec/validate_drp.AM1.design" ).json()
spec
Out[9]:
{'name': 'validate_drp.AM1.design',
 'threshold': {'unit': 'marcsec', 'value': 10, 'operator': '<='},
 'tags': ['AM1', 'design', 'achromatic'],
 'metadata_query': {}}

Note that the operator in the lsst.verify specifications is such that “measurement op spec” is True if the measurement passes the specification. But we need the opposite when configuring alerts. In order to do that we created this mapping:

In [10]:
inverse_operation = { '>': '<=', '>=': '<', '<': '>=', '<=': '>'}

Also note that triggers do not support equality and inequality, so if converting a lsst.verify spec to a Honeycomb trigger and the spec uses an equality match, you will have to decide whether the appropriate trigger comparison is greater than or less than the value.

In [11]:
trigger = {"name": "AM1 alert",
           "query": query,
           "threshold": {"op": inverse_operation[spec["threshold"]["operator"]],
                         "value": spec["threshold"]["value"]},
           "frequency": 1800
          }
trigger
Out[11]:
{'name': 'AM1 alert',
 'query': {'breakdowns': ['dataset', 'filter_name'],
  'calculations': [{'column': 'validate_drp.AM1', 'op': 'AVG'}],
  'filters': [{'column': 'dataset', 'op': '=', 'value': 'validation_data_hsc'},
   {'column': 'filter_name', 'op': '=', 'value': 'HSC-R'}],
  'filter_combination': 'AND'},
 'threshold': {'op': '>', 'value': 10},
 'frequency': 1800}
In [12]:
r = requests.post("https://api.honeycomb.io/1/triggers/squash-demo", json=trigger, headers=headers)
r
Out[12]:
<Response [201]>

Finnaly, all Triggers in a dataset may be retrieved by:

In [13]:
r = requests.get("https://api.honeycomb.io/1/triggers/squash-demo", headers=headers)
r.json()
Out[13]:
[{'name': 'AM1 alert',
  'frequency': 1800,
  'query': {'breakdowns': ['dataset', 'filter_name'],
   'calculations': [{'column': 'validate_drp.AM1', 'op': 'AVG'}],
   'filters': [{'column': 'dataset',
     'op': '=',
     'value': 'validation_data_hsc'},
    {'column': 'filter_name', 'op': '=', 'value': 'HSC-R'}]},
  'threshold': {'op': '<=', 'value': 10},
  'recipients': [{'type': 'email', 'target': 'alerts@example.com'}],
  'id': 'i4WqTioppHA'},
 {'name': 'AM1 alert',
  'frequency': 1800,
  'query': {'breakdowns': ['dataset', 'filter_name'],
   'calculations': [{'column': 'validate_drp.AM1', 'op': 'AVG'}],
   'filters': [{'column': 'dataset',
     'op': '=',
     'value': 'validation_data_hsc'},
    {'column': 'filter_name', 'op': '=', 'value': 'HSC-R'}]},
  'threshold': {'op': '<=', 'value': 10},
  'recipients': [{'type': 'email', 'target': 'alerts@example.com'}],
  'id': 'wXbfYt59QZE'},
 {'name': 'AM1 alert',
  'frequency': 1800,
  'query': {'breakdowns': ['dataset', 'filter_name'],
   'calculations': [{'column': 'validate_drp.AM1', 'op': 'AVG'}],
   'filters': [{'column': 'dataset',
     'op': '=',
     'value': 'validation_data_hsc'},
    {'column': 'filter_name', 'op': '=', 'value': 'HSC-R'}]},
  'threshold': {'op': '>=', 'value': 10},
  'id': 'k5tTFvd9vyd'},
 {'name': 'AM1 alert',
  'frequency': 1800,
  'query': {'breakdowns': ['dataset', 'filter_name'],
   'calculations': [{'column': 'validate_drp.AM1', 'op': 'AVG'}],
   'filters': [{'column': 'dataset',
     'op': '=',
     'value': 'validation_data_hsc'},
    {'column': 'filter_name', 'op': '=', 'value': 'HSC-R'}]},
  'threshold': {'op': '>=', 'value': 10},
  'id': '3HwsMiwtmu1'},
 {'name': 'AM1 alert',
  'frequency': 1800,
  'query': {'breakdowns': ['dataset', 'filter_name'],
   'calculations': [{'column': 'validate_drp.AM1', 'op': 'AVG'}],
   'filters': [{'column': 'dataset',
     'op': '=',
     'value': 'validation_data_hsc'},
    {'column': 'filter_name', 'op': '=', 'value': 'HSC-R'}]},
  'threshold': {'op': '>', 'value': 10},
  'id': 'xrHwYp1vPE'}]