Build and Gated Check-in

1. Build and Gated Check-in

2. Goals

Develop understanding of:
• Build system architecture
• Differences between AX6.x, and AX7
• Current system state
• Roadmap
• Common operations scenarios

3. Knowledge

• Process FAQ - http://fargobuild/processfaq/
• AX FAQ - http://fargobuild/axfaq/
• SharePoint sites:
ESS Ops
Build Infrastructure
Dev Infrastructure
ESS Documents
• Happydev - http://happydev

4. Overview

Source Control
BB
Execution
Build Reporting – DBs / Services
Checkin
Wizard
BB
Wizard
Code
Review
Build
Tracker
Queue
Processor
BuildTracker
Checkin Infrastructure
Gate execution
Official build Execution
Eng.
Fundamentals
Developer makes a change
SCRAM
SCRAM
WTT
WTT
Lab Services
CodeFlow
Enlistme
CoreXT
PRSS
QE, SCA,
SymIndex
BGIT

5. Build Central Infrastructure

6. Lab Footprint

DCSRDCHKINFSA
Build Drops
BuildTracker
Gates
Code review pack
share
Dcsrdchkinfs01
RR1SQLB02
dcsrdbldfsa
Dyn
Log/Pack share
RDVMHL1109-05
Hosts Dyn DFSN
dcsrdblddrop03
RR1SQLB03
DCSRDBTRACKER01
Queue Processor
Synthetic checkin
service
DCSRDGTDSYMA
Dcsrdgtdfs01
BuildService
Log Shares
Reporting parsers
Tools share
Pending pack share
Machine reporting
Status emails
Schedules
maintenance tasks
Symbol file share
DCSFRFargoBuild
Hosts FAQ sites
DCSRDRPTSQLA
DynamicsAXBuilds
FAQData
ProcessFAQ
DCSRDBLDIIS3
Hosts Branch Config site
Hosts Autointegrate site
DCSRDBLDSQL01
AutoIntegration
Dcsredbbcfs01
Statistics
UnitTests

7. Source Depot - Overview

Server Info:
Hosted/Managed by BGIT
Connect To - vedbuitfsa06:4020
Actual Server - TK5SDSERVDAX01
What to do when you notice the depot
is down

8. Source Depot - Overview Contd.

Proxies
Enhance local user experience
Both BGIT-managed and private
Current proxies : “sd proxies -o”
14 exist today
Administration
• //depot/Admin/…
• SD Admin Explorer - http://toolbox/22929

9. Source Depot - Overview Contd.

Help
• Sd help
• Sd help undoc
Protections
• Controls who has permission to what
• Modifying protections
Triggers
• Logic to run on command execution
• Main usage – preventing direct submits
• Modifying triggers

10. Source Depot - Overview Contd.

Maintenance
• BGIT-managed updates to server/SD instance
• Remove unused clients
Client components
sd.exe
sdapi.dll
sdb
sdv
Packing tools: sdp, jjpack

11. Source Depot - Branching

Main
Source Depot - Branching
BuildServices
Purpose:
Dormant/
Integration
DAX5
DAX5SP1HF
DAX6SE
DAX6HF
DAX6HF_STAB
DAX62Stab
DAX63SE
DAX63HF
DAX63HFSTAB
RainINT
DAX63NS
DAX63NSStab
DAX6_BT
DAX6
DAX62OOBSTAB
DAX62OOB
DAX62
In-Market
DAX62HFSTAB
DAX62HF
LCS
DAX62SE
DAX62CD
AX7
RainMain
Infra
RainMainStab
RainFND
RainStab
RainLCS
RainLCSV5
App<Rel.>
RainFNDStab
RainGL
RainLCSStab
LCS
RainLCSHF
LCSStab
Plat<Rel.>
CoreServices
LCSHF

12. Source Depot - Integrations

Introduction
…it is “to propagate changes happened to a namespace (source namespace)
during a certain period of time into another namespace (target namespace)”.
-SD Integrations Guide
Four States to Consider:
1.
2.
3.
4.
State of the source just before a given period of time
State of the source exactly at the end of a given period of time
State of the target at the moment of integration
State of the target at the end of the integration

13. Source Depot - Integrations Contd.

Revision
• Versioning at the file level
• Referenced with ‘#’
• Example – sd sync //depot/rainier/rainfnd/foo.txt#2
Changelist
• Collection of file revisions commited at a single point in time
• Referenced with ‘@’, -c <changelist>, <changelist>
• Example – sd describe 123456
Label
• Collection of changelists
• Referenced with ‘@’
• Example – sd sync …@7.0.123.0

14. Source Depot - Integrations Contd.

SD Integrate
• Forward Integrate (FI) versus Reverse Integrate (RI)
• Full branch integration
• sd integrate -b <child branch> @<revision>
• Point integration
• Sd integrate -b <child branch> @<rev 1>,<rev 2>
• Manual mapping - renaming or baseless integrates
• sd integrate //depot/b1/foo.txt#1,2 //depot/b2/foo.txt

15. Source Depot - Integrations Contd.

SD Resolve
Automatic resolve – sd resolve -as
Manual resolve – sd resolve
Accept Thiers (at) vs. Accept Yours (ay) vs. manual merge (e)
Conflict/Manual resolution:
>>>> ORIGINAL sources.all#2 ++++++++++++++++++++++++++++++++++++++++++++++++
#endif
MSBUILD_VERSION=4.0
==== THEIRS sources.all#3 ++++++++++++++++++++++++++++++++++++++++++++++++
#endif
==== YOURS sources.all ++++++++++++++++++++++++++++++++++++++++++++++++
#endif
MSBUILD_VERSION=12.0
<<<< END ++++++++++++++++++++++++++++++++++++++++++++++++

16. Source Depot - Integrations Contd.

AutoIntegrate tool
• Set of automation wrapping “sd.exe integrate”
• Custom resolvers
• MSBuild custom tasks/targets
• //depot/../Build/AX_Tools/AutoIntegrate/...
• Web Resolve
• Run via command line “Integrate” command or BuildTracker jobs

17. Source Depot - Pack Files

Jjpack.exe
• Primary packing format used in build/gates
• Supports integrations
• Does not support ‘undo’ – downgrades file op
SDP.exe
• Does not support integrate options
• Double-click diff
• Used more commonly by Engineers to pass around changes

18. BuildTracker

Our Instance:
• http://daxbuild/BuildTracker
BT product site:
• http://buildtracker
BT Lab Solutions DG:
• btracker

19. BuildTracker - Services

RR1SQLB02
• Database
RR1SQLB03
• BuildA – Access Management
• BuildK – Manages BT encryption key
• BuildL – Lab Monitoring (WMI monitoring)
• BuildM – Mail and notifications
• Web UI – http://daxbuild/buildtracker
• API - net.tcp://RR1SQLB03:9700/BuildTrackerApi
DCSRDBTRACKER01
• BuildS – Schedules job queues
Client machines
• BuildQ – Worker service

20. BuildTracker - Lab Maintenance

Build Team Services Project
Windows update on clients
Reports
DB Backups
Auto-archive and drop cleanup
Gate Queue Processor check
BT Automation rules

21. BuildTracker - Lab Maintenance Contd.

Build Tracker Automation Library
• Rules framework for enforcing consistent
practices in our BT instance
• Runs daily at 3am PST
• Source in BuildServices at
\build\source\MS.Dynamics.Build\...

22. BuildTracker - BT.exe

• Wrapper for common lab maintenance and reporting functions using
BuildTracker’s APIs
• Available functions differ between BuildServices and product
branches
• Source - \build\ax_tools\buildtracker\bt\...

23. AX7 / Rainier

24. The Product

• CTP Technical Concepts guide on HappyDev

25. The Product

• CTP Technical Concepts guide on HappyDev

26. Lab Footprint (AX7)

DCSRDCHKINFSA
Code review pack
share
Build Drops
Test
dcsrdbldfsa
Test Clients
dcsrdblddrop03
Symbol file share
Hosts Dyn DFSN
BuildTracker
DCSRDBTRACKER01
24 machines
2 CPU cores
16GB Memory
Primitive Clients
14 machines
Build Clients
Dyn
Dcsrdchkinfs01
Log/Pack share
104 machines
DCSRDGTDSYMA
Gates
VMs allocated via Freebird
8 CPU Cores
60GB Memory
200 + 500 GB FusionIO-backed
2 CPU Cores
16GB Memory
Stress Clients
7 machines
4 CPU Cores
16GB Memory
Dcsrdgtdfs01
150 machines
VMs allocated via Freebird
8 CPU Cores
48GB Memory
200 + 500 GB FusionIObacked
Allocated 80% primary gate
processing / 20% build leg
RDVMHL1109-05
RR1SQLB02
BuildService
Log Shares
Reporting parsers
Tools share
Pending pack share
Machine reporting
Status emails
Schedules
maintenance tasks
Gate Clients
Queue Processor
Synthetic checkin
service
LKG Service
RR1SQLB03
DCSRDRPTSQLA
DCSFRFargoBuild
Hosts FAQ sites
DynamicsAXBuilds
FAQData
ProcessFAQ
DCSRDBLDIIS3
Hosts Branch Config site
Hosts Autointegrate site
DCSRDBLDSQL01
AutoIntegration
Dcsredbbcfs01
Statistics
UnitTests

27. Ax7 / Rainier Branches

RainINT
DAX62STAB -> DAX62 -> DAX6 -> DAX6_BT -> Main
RainMain
BuildServices
RainMainStab
RainFND
RainStab
RainLCS
RainLCSV5
App Servicing
Branches (e.g.
AppCTP8,
AppRTW)
RainFNDStab
RainGL
RainLCSStab
LCS
RainLCSHF
LCSStab
Platform
Servicing
Branches (e.g.
PlatCTP8,
PlatRTW)
LCSHF
CoreServices

28. CoreXT - Introduction

What is it?
A layer of rules, processes, features and automation
A common set of tools for developers, testers, builders and labs alike.
Generic and decoupled from the applications being built
Community effort within Microsoft
Existed since 2000 – estimate is used by 1/3 of the company
Latest release – v5.0
More information – http://corext

29. CoreXT - Our Environment

Overview
Current version – v3.2
Build.exe to each project node
Dependency Based Build (DBB) enabled
Tracer/Analyzer disabled
No adoption of CoreXT integration targets/properties
CoreXT V5 package manager backported

30. CoreXT - Environment Setup

Entry point
• %inetroot%\tools\path1st\myenv.cmd
• Shortcut by:
• Open cmd.exe
• Navigate to enlistment root
• Run “setenv”
Branch-specific configurations
• %inetroot%\build\branchcfg\myenv_<branch>.cmd

31. CoreXT – Package Management

• Branch package dependencies defined in a checked-in package
manifest
• On CoreXT startup, missing packages are retrieved and installed into a
local cache directory.
• Build processes consume package contents from the local cache
• More information on HappyDev

32. CoreXT - Externals

Public Externals
• ANY external reference or toolset
• ANY internal reference or toolset submitted as a binary
• Location – NuGet package repository (previously: %inetroot%\public\ext\...)
Toolsets
• Often accompanied by makefiles/targets which integrate into CoreXT
Approvals
• True external libraries (both Non-Dynamics Microsoft produced or 3rd party)
need to receive LCA approval before check-in.
• External Components Search

33. CoreXT - Build

<Root>
Build.exe
• Windows build tool
• Orchestrator for building/traversing
the build graph
• Our version – 6.2.8427.0
• ‘dirs’ and ‘sources’
• amd64 is default build target
Dirs
Dirs
Dependency Based Build (DBB)
• Build node dependencies drive build
graph traversal
• Sources.dep
Sources
Sources
Makefile.def
Makefile.def
MSBuild
WiX
Dirs
Sources
Sources
Makefile.def
Makefile.def
CMD
VC++

34. CoreXT - Build Contd.

PASS[0|1|2]
• PASS0 – Generation of files, copying of static content
• PASS1 – Compilation, X++, C#, C++ (both compile and link) all happen here
• PASS2 – Packaging/Binplacing/Setup
• Assembling the content from PASS0 and PASS1 into their final containers
• PASS2_BINPLACE – binplacing of built bits (mostly from PASS1)
Project Inclusion
• Integrating your project into CoreXT’s build
• Dirs, sources, sources.dep, makefile, makefile.inc

35. CoreXT - Build Contd.

Rainier Build Topics
XPPC
Proxies/FormAdaptors
DBSync
Packaging

36. CoreXT - MSBuild

Current Version – v14.0
Build.exe to MSBuild Handoff
• \public\ext\tools\msbuild.def inclusion in project makefile
• \tools\path1st\msbuild.cmd wraps msbuild.exe
• MSBUILD_PROJECT in sources
NMake
Build.exe
Makefile.def
Msbuild.def
MSBuild.cmd
MSBuild.exe
My_Project.csproj

37. CoreXT - FxCop

Source:
• All Branches: %inetroot%\public\ext\fxcop\...
Runs on:
• Binplace via %inetroot%\tools\build\buildtypes\fxcop_build.cmd
• MSBuild project when RunCodeAnalysis=true
• Nothing currently enforcing correct ruleset here
Relevant Sources Macros:
• Specify additional parameters - FXCOP_USER_FLAGS
• Skip FxCop on Binplace - BINPLACE_FLAGS=--no-postbuild-fxcop

38. CoreXT - FxCop Contd.

Rule Categories
• SWI/Stock – rules fulfilling QE requirements or shipped with FxCop
• Example – Require CultureInfo be specified when calling a member that accepts it
• Dynamics source - enforce Org-specific patterns
• Example - Format of Copyright on binary’s properties
• Dynamics Test – enforce patterns on test code
• Example – Enforce inclusion of required test attributes on test cases

39. CoreXT - FxCop Contd.

Common Problems
Missing reference assembly
Multi-proc/race condition
Deadlock/timeout
Custom rule crashes
Logs
• XML format per scanned assembly
• From Build log root – FXCopLogs%BuildType%
FxCop error codes - http://msdn.microsoft.com/en-us/library/bb429400(v=vs.80).aspx

40. CoreXT - Customization

Stock CoreXT
• Most source scripts under \tools\...
• Makefiles under \public\ext\tools\...
• As a rule don’t modify stock CoreXT
Customization points
\build\automation\...
\build\path1st\...
\build\buildtypes\...
\build\sources.all
\build\preenv.cmd
\build\myenv.cmd

41. CoreXT - Automation

CoreXT + BuildTracker
• Stock automation scripts - %inetroot%\tools\build\automation\...
• Customized automation scripts - %inetroot%\build\automation\...
bldTree.cmd
bldCommon.cmd
bldDone.cmd
bldStart.cmd
bldShare.cmd

42. CoreXT - Codereview

• Wizard based on AutoProcess console that wraps CodeFlow
submissions
• Binaries\Configuration - \build\CodeReview\...
• Source - \build\source\checkinwizard\...
• Managed pack share - \\dcsrdchkinfsa\CodeReviews
• Packs retained for ~2 years.
• Command: “Codereview -c <local changelist>”

43. CoreXT - DynCop

• Home-grown static analysis tool
• Developed and used initially for CoreXT v3.2 upgrade
• Develop rules which enforce consistency and best practices across the
build environment
• Executes locally, in gates, and LMC
• Source – VSTS DynCop Git repo
• Documentation on HappyDev
• Product teams adopting for enforcing practices in their teams

44. CoreXT – Platform-As-A-Binary

• Historically, AX has been a monolithic codebase
• Re-architect product into Platform component and Application component
to be build/deployed/validated/released/serviced independently
• Platform and Application are exchanged with one another as Last Known
Good (LKG) zip packages
• Further work expected to the product and ESS infrastructure supporting
this
• More on LKG Management here

45. CoreXT - Dirs Filtering Framework

• Mechanism used to partition codebase for initial
Platform/Application build separation
• Uses build.exe’s filtering on dirs {tokens}
• Tool source - \build\ax_tools\DirsFilterManager\...
• Executable - \build\ax_tools\bin\DirsFilterManager.exe
• DynCop rules enforcing correct tokens
• HappyDev documentation

46. CoreXT - Dirs Filtering Framework Contd.

• Examples:
Build retail –BuildOnly:fnd
Build retail –BuildConfig:Platform
Build retail –ExcludeTokens:app
Source
DIRS= \
AppIL \
Kernel \
AppIL
Kernel
DIRS= \
Foo{!app} \
Bar{app}\
Foo
DIRS= \
Zoo{fnd} \
Bar
Zoo

47. CoreXT - Odds and ends

Aliases
• Maps a command to an alias
• Example – ‘codereview’ invokes the script to launch the code review wizard
• Specified globally per branch in %inetroot%\build\aliases\aliases.pub
PrivateDev
• Global private directory
• Include //depot/private/privatedev/developer/<alias>/… in your client spec
• Map to %inetroot%\private\developer\<alias>
• Private aliases
• Add a file \build\aliases\aliases.pub under private directory

48. Gates

49. Gates - Technologies

WTT
• Workflow engine
• Machine
management/maintenance
Powershell
• Log parsing
• Reporting
• Queue processor
Managed code
Log parsing
Reporting
Scheduling
Orchestration
Monitoring
Wizards
Web Service
Cmd
• (nearly) All of the above

50. Gates - 50k Feet

Pre Checkin Checks (WTT)
Checkin Wizard
User Submits change
Queue Processor
Available machine found.
Checkin Scheduled
Gate Run (WTT)

51. Gates - Submitting

Checkin submit -c
123456
Launch Checkin
Wizard
User chooses
changelist files
Checked-in gate
spec XML
Analyze
changelist
User
configures run
(chooses to
run or bypass
optional tasks
User chooses
TFS or PS bugs
Checkin queue
entry
User submits
checkin
End Wizard
Pack file
Checkin Spec XML

52. Gates - “Checkin” Command

Usage: checkin <operation> <operation parameters>
Operation
Description
Submit -c <changelist>
Initiate a new checkin. Launch the wizard
Resubmit -id <checkin id>
Resubmit a previous checkin
Abort -id <checkin id>
Abort a current running checkin
Diff -id <checkin id>
Diff the changes for that checkin
Apply -id <checkin id>
Apply the changes from a checkin locally
Repack -id <checkin id> -c <changelist>
Repack a previous checkin’s pack with the contents of the
specified local changelist
Prioritize -id <checkin id>
Prioritize the checkin in the queue
Bypassqueuelock -id <checkin id>
Allow a checkin to run even if the gates are closed for that
branch
Source script: \public\ext\tools\x86\checkin.cmd

53. Gates - Checkin Wizard

• Wizard for submitting to the gates
• Based on AutoProcess console framework
• Configuration: \build\CheckinWizard\...
• Source: \build\source\CheckinWizard\...
• Command: checkin submit -c <changelist>

54. Gates - Queue Processor

Core
• Orchestrates scheduling of requested check-ins
• Scheduling dimensions:
• Machine availability
• Blocking files
• Branch lockdown
• Main script is written in Powershell
• Cmd wrapper manages logs and invokes core PS script
• Logs retained for 24 hours
• Executes every 5 minutes as a schedule task on gate server

55. Gates - Queue Processor Contd.

Core Contd.
• BuildServices branch - %inetroot%\build\wtt_checkin\SetExternalReferences.cmd
• Log location - %QueueProcessLogDirectory%
• Execution server - %QueueProcessServer%
• BuildServices -
%inetroot%\build\wtt_checkin\QueueProcessor\...
BBCheck.exe
• Core scheduling utility to the gate WTT pool
• Wrapped by “checkin” command
• Abort, Submit, Resubmit, Status, Diff, Apply, BypassQueueLock, Prioritize, Repack, debug
• All branches - %inetroot%\build\source\BBCheckSystem\bbcheck\...

56. Gates - Queue Processor Contd.

Merge Resolution Check
• Every 15 minutes for branches that have queued, blocked checkins
• 3-way merge attempted on all files involving:
• Depot state of the file
• State of file in checkin blocked on
• State of file in checkin that’s blocked
• 0 conflicts on all blocked files unblocks checkin
• All branches - %inetroot%\build\wtt_checkin\MergeCheck\...
QueueParse.exe
• Part of ReportingParser cycle
• Reflects blocking file state into DB
• BuildServices - %inetroot%\build\source\QueueParse\...

57. Gates - Execution

Checkin is Scheduled and Executes
Machine allocation/Preparation
(Sync, nuke, uninstall old deployments, apply changelist)
Changelist and checkin
validations
Build
Deploy and Test
Post Steps
(kernel, app, test, debug flavors)
(BVTs, UTs, BP checks)
(Determine Results, Submit CL, Send report email, copy logs,
update bugs, teardown and release machine)
(DynCop,
SDDump
macro
Schedule
Build leg
validators, code coverage
pairing , etc.)
Primitives
Stress
BVT
Return results
Initiate parallel test runs
Schedule Run
Preparation
Pre Checkin
Retail Build (Platform or
Application)
Post-Build Merge
C# UTs
Deploy
X++ CITs
Determine Results
Schedule parallel leg builds
Return results
Build Leg Machine
Debug Build (Platform or
Application)
Determine Results/
End

58. Gates - Execution Contd.

Primary Machine
Maintains checkin state
Executes retail build
Schedules leg builds and SCRAM runs
Deploys
Executes unit tests (C# and X++)
Evaluates task results
Submits (or rejects) changelist
Sends alerts
Updates bugs
Hosts checkin build share
• Retained for up to 3 days
• \\<machine name>\CheckinBuilds\<checkin ID>\

59. Gates - Execution Contd.

Legs
• Build leg
• Applies change
• Executes a build
• Reports result back to primary machine (via file share)
• Test leg (SCRAM)
• Runs in SCRAM infrastructure
• Consumes build hosted on primary machine
• Returns result summary to primary machine (via SCRAM.exe/XML file)

60. Gates - Execution Contd.

Environment
• %binFolder% - %inetroot%\build\wtt_checkin
• %workFolder% - %inetroot%\build\wtt_checkin\work
• %parm<parameter name>% - Value from WTT job parameter
• %workFolder%\WTTFlag.tmp – indicates running in the gates
• %binFolder%\SetExternalReferences.cmd
• Sets things like %StatisticsDatabaseServer%, %QueueProcessorServer%, etc.

61. Gates - Tasks

Task Intro
• Single script under %binFolder%
• Invoked from WTT job task
Format
Set up gate-specific variables (includes getting WTT job parameters)
Log task start in [Statistics].[dbo].[GateTask]
Determine if task can run (disabled, pre-req missing, aborted)
Task setup – E.g. restoring DBs, stopping/starting services
Task logic – E.g. running “build” command
Evaluate result – Write as semaphore file to %workFolder%
Finalize the task in [Statistics].[dbo].[GateTask]
Update task detail in [Statistics]
Exit

62. Gates - Tasks Contd.

Primary Machine
Preparation
Sync
Apply Changelist
Pre-Checkin Checks
Pre-Build Preparation
Build Retail-platform (if applicable)
Build Retail-application (if applicable)
Post-Build
Schedule Tests
Deploy
MSTest Tests (C# UTs)
Xref-selected tests (if applicable)
Unit tests (X++ CITs)
* Wait
Submit
Bug Update
Log Copy
Dev gate Leg
Preparation
Sync
Apply Changelist
Build Debug-platform
Build Debug-application
Log Copy
SCRAM
• BVT
• Primitive Tests
• Stress Tests

63. Gates - Tasks Contd.

Adding a task
Create a task script
Create it under %inetroot%\build\wtt_checkin\<task name>.cmd
Add the task to the WTT job (usually requires a new job version)
Update determine results to evaluate if the task ran, passed or failed
Create the task code in [Statistics].[dbo].[GateTaskCode]
Create the task exit codes in [Statistics].[dbo].[GateTaskExitCode]
Update GCM and DB to include configuration of the task.
Create the shell for GateTask to execute it locally (create it under
%inetroot%\build\wtt_checkin\<task name>FE.cmd)
• Update GateTask.cmd to include the option and execution for the task

64. Gates - Fast Build

Gates - Symbol/binary indexing
• Every gate build indexed to http://symweb
• Deduplication and filtering of binaries (RemoveDuplicates.exe)
• Location set in branch config as %GateSynShare%
• Process invoked in Post Build
• Build\wtt_checkin\CheckinBuildDrop.cmd
• Space-based retention handled by the file share
• Additional Documentation here

65. Gates - Fast Build Eligibility

Gates - Synthetic Checkins
• Service scheduling regular test checkins to assess the stability of a
branch/checkin path
• Windows service hosted on RDVMHL1109-05
• Source in BuildServices branch at \build\source\SyntheticCheckinService
• Current flavors:
• SyntheticCheckin – default type, executes like a typical checkin in the branch
• BuildOnly – executes a checkin that just does a build in a dedicated pool
• Differentiated in reporting on [Statistics].[Stats].[SyntheticType]

66. Gates - Fast Build Execution

Gates - Reporting / Alerts
• MBSUSP
• MBSRPT
• Email
Status
Volume
Build Breaks
Offline Machines
Queue Cycle Report
…many many more…

67. Gates - Fast Build : FastApp

Gates - Statistics DB / BuildService
Statistics
Primary repository of gate execution and configuration data
Lives on dcsredbbcfs01
FAQ on common tables
Exposed via BuildService (for new development)
BuildService
WCF Service
Expose gate data from [statistics], [unittests]; build data from [dynamicsaxbuilds]
Source in BuildServices branch - \build\source\BuildService
Hosted on dcsrdgtdfs01

68. Gates - Fast Build : NightlyBootstrap

Gates - Development
Task Source
• %inetroot%\build\wtt_checkin\...
Tools Source
• %inetroot%\build\ax_tools\...
• %inetroot%\build\source\...
• %inetroot%\build\scripts\...
WTT Artifacts
• Checkin jobs - $\GatedCheckin
• Utilities/Maintenance - $\GatedCheckin\Utilities

69. Gates - Symbol/binary indexing

Gates - Development Contd.
Maintenance and core tools
• BuildServices branch
• %inetroot%\build\wtt_checkin\MachineConfiguration\...
• %inetroot%\build\wtt_checkin\
Test environment
• Run locally using “gatetask” command
• Redirect to private WTT pools
• Create new synthetic checkin type

70. Gates - Synthetic Checkins

Gates - Monitoring / Analysis
Gated Checkin Monitor (GCM)
• Gate/build monitoring dashboard
• Quick access to common actions (e.g. Resubmit, diff, incident creation)
• Gate configuration
• Queue processor by branch
• Gate tasks by branch
• Source in ‘BuildServices’ branch
• %inetroot%\build\source\GatedCheckinMonitor\...
Logs
• Console log vs. build.exe vs. detail log

71. Gates - Reporting / Alerts

BuildTracker Managed Builds

72. Gates - Statistics DB / BuildService

BuildTracker Builds - Official Builds
Purpose
• Produce a ship-quality build
• All infrastructure in place to be ‘All On’
• Provide a rollup of changes to validate against
Cadence
• Runs daily with sporadic OOB runs
• Kick-off staggered across branches
• RainMain* – 9pm PST
• RainFND* – 10pm PST

73. Gates - Development

BuildTracker Builds - Buddy Builds
Purpose
• Provide engineers with a private build containing their pending changes to be used in
pre-checkin validation.
Cadence
• On-demand/user initiated
• Initiated either directly from the BuildTracker web UI or by running “buddybuild –c
<changelist>” in CoreXT
• Permissions controlled by groups:
• redmond\bbscramu
• europe\axsrcacc_ram
Resources
• Jobs– SCRAM Buddy Build-Redmond – Platform, SCRAM Buddy Build-Redmond Application
• Machine pool – Redmond – Rainier Buddy Build
• Drop share - \\dcsrdblddrop03\Build1\BuddyBuilds\Rainier
• Retention is space-based and managed external to BuildTracker

74. Gates - Development Contd.

BuildTracker Builds - Buddy Build Wizard
• Binaries/Configuration - %inetroot%\build\BBCheckinWizard
• Source locations
• Source control grid (first screen):
build\Source\CheckinWizard\src\CheckinComponents\Generic\
• Buildtracker screen (second screen):
build\BTCheckinWizard\ExternalContributors\Produc\
• Managed pack share
• Location - \\dcsrdblddrop03\BuddyBuildPackages
• Retention – 30 day policy

75. Gates - Monitoring / Analysis

BuildTracker Builds - Rolling Builds
Purpose
• Provide faster build/validation cycle than nightly official builds
Cadence
• RainFND Rolling Master
• 4x per day
• Initiates PDL runs per build

76. BuildTracker Managed Builds

BuildTracker Builds - Jobs
Main Build
Buddybuild
Localization (Static)
Retail Build
(Platform or
Application)
Rolling Platform Build
Localization (Static)
Debug Build
(Platform or
Application)
Retail Build
(Platform or
Application)
Localization (Static)
Debug Build
(Platform or
Application
Retail-Platform Build
AxPackage_Contoso
AxPackage_Contoso
Code Sign (CD and
Services)
BVT
TSImport
AxPackage_Contoso
Generate BacPac
Generate BacPac
Translation-submit

77. BuildTracker Builds - Official Builds

BuildTracker Builds – Master Builds
Master
Main Build
Post Build Tests
LMC
DevFabric BATs
X++ Tests - Full
Stress Tests
Managed flavors
APIScan
FxCop
DynCop
ModernCop
C# Unit Tests
Primitive Tests
BVT Stabilization
LKG Compatibility
BinScope
CodeSign
Check
BinSearch
PoliCheck
Rolling Master
Rolling Platform Build
Post Build Tests
PDL
Primitive Tests
Stress Tests
• Collection of child jobs to execute as a single job
instance
• Typically include a build job and some post build
activities to run if the build was successful
• Usually what daily schedules are set to kick off

78. BuildTracker Builds - Buddy Builds

CodeSign
• Authenticode versus Strong Name
• Signing process
• Delay signing
• Authsign.exe
• Certificates
• Non-standard formats/processes
CSPKG
APPX
ClickOnce/VSTO
VSIX
Packages (general)
Links:
• CodeSign – http://codesign.gtm.microsoft.com/
• CodeSignInfo – http://codesigninfo/

79. BuildTracker Builds - Buddy Build Wizard

Localization
LSBuild
• Command line build tool to create localized product files
• Uses English version of file as input
• Consumes LCL files provided by MBSI
• %inetroot%\source\translation\LCL
• Use various parsers to determine what to do
• Only one custom one – KTD files

80. BuildTracker Builds - Rolling Builds

Localization - Contd.
PseudoLoc
• Simulated localized content
• Intended to catch localization problems as early as possible
• Pseudoloc examples from AX 2012 R3
Build Legs
• Localization-initialize – copies drop\translation to release share
• Localization-server – Builds localized KTD files
• Retail-Platform - \source\Platform\Localization\GeneratedPASS2
Retail-Application - \source\net64resources
• Build localized resource files
• %inetroot%\build\localization\localization.binaries.config.xml
• Translation-submit – Signs and submits localized files to \source\translation\

81. BuildTracker Builds - Jobs

QE / LMC
Quality Essentials (QE)
• Set of validations focused on ensuring overall release quality
Regulatory/legal
Industry standards
Customer Experience
Security
Consistency with Microsoft Engineering Policies
• More information - http://qe/
Last Minute Checks (LMC)
• Executes QE and Org-specific build quality checks
• Example from DAX62

82. BuildTracker Builds – Master Builds

Post-ship activities
• Local and long-term backup
• Source code archive
• Symbol index/archive

83. CodeSign

Visual Studio Online (VSO)
• What is VSO?
• Our instance: http://msdyneng.visualstudio.com
• VSO-hosted
Source control via Git
Work item tracking
Build definitions
Build requests/scheduling/reporting
• On-Premises
• Build controller/agent – RBLD0071AC-TFS
• Build drops - \\DCSRDBLDDROP04\Build1\Drop

84. Localization

In-Market

85. Localization - Contd.

AX6.x Product
• AX 2012 Technet site
• AX 2012 Architecture
• AX 2012 MSDN Developer site

86. QE / LMC

AX6.x Product Contd.
AX32.exe – Win32 client
AX32Serv.exe - Server component running as a Windows Service
MorphX – development environment
Model store – database storing the P-code representing the application
inside the platform
• AXDBBuild_Model – build-time instance name
• AXDBDev_Model – runtime instance name build/gates/deploy.cmd
• AXDB – common runtime instance name on other lab machines
• Business database – application database housing the data model for
business logic
• AXDBDev – runtime instance name

87. Post-ship activities

Lab Footprint (In-Market)
Build Drops
Gates
dcsrdblddrop02
dcsrdbldtempfs1
dcsrdblddrop03
dcsrdbldtempfs2
dcsrdbldfsa
BuildTracker
RR1SQLB02
RR1SQLB03
Dcsrdchkinfs01
Dcsrdgtdfs01
Build Clients
~50 machines
Log/Pack share
VMs allocated via Freebird and
UIP-managed physical machines
2-16 CPU Cores
8-32GB Memory
250GB – 2TB enlistment drives
BuildService
Log Shares
Reporting parsers
Tools share
Pending pack share
Machine reporting
Status emails
Schedules
maintenance tasks
Gate Clients
110 machines
VMs allocated via Freebird
and UPI-managed physical
machines
2-16 CPU Cores
8-32GB Memory
250GB – 2TB enlistment
drives
Allocated 80% primary gate
processing / 20% build/test
leg
RDVMHL1109-05
Queue Processor
Synthetic checkin
service
DCSRDCHKINFSA
Code review pack
share
DCSFRFargoBuild
Hosts FAQ sites
Dyn
Hosts Dyn DFSN
DCSRDRPTSQLA
DynamicsAXBuilds
FAQData
ProcessFAQ
DCSRDBLDIIS3
Hosts Branch Config site
Hosts Autointegrate site
DCSRDBLDSQL01
AutoIntegration
Dcsredbbcfs01
Statistics
UnitTests

88. Visual Studio Online (VSO)

In-Market Branches
Main
BuildServices
DAX5SP1HF
DAX6SE
DAX6HF
DAX6HF_STAB
DAX63SE
DAX63HF
DAX63HFSTAB
DAX63NS
DAX63NSStab
DAX6_BT
DAX6
DAX62OOBSTAB
DAX5
DAX62OOB
DAX62
DAX62HFSTAB
DAX62HF
DAX62SE
DAX62Stab
DAX62CD

89. In-Market

Servicing Concepts
• Dependency toolset
• Hotfix
• Binary vs. Application
• Rollup
• Cumulative Update (CU)

90. AX6.x Product

CoreXT - Our Environment
Overview
• Current version – v1
• Build.exe to each project node
• SYNCHRONIZE_BLOCK, SYNCHRONIZE_DRAIN and dirs file ordering used to
drive build ordering
• Custom ‘mydirs’ logic used to partition the build tree corresponding to
custom build types
• Binplace, Placefiles/miscfiles both supported

91. AX6.x Product Contd.

CoreXT - MSBuild
• Current Version – v4.0
• Build.exe to MSBuild Handoff
• Automatically included in all project makefiles via \build\makefile.inc
• \build\makefiles.msbuild creates a response file (msbuild.rsp) in the
project directory and invokes msbuild.exe directly.
• MSBUILD_PROJECT in sources
NMake
Build.exe
Makefile.def
Makefile.inc
Makefile.msbuild
MSBuild.exe
My_Project.csproj

92. Lab Footprint (In-Market)

CoreXT - Build
<Root>
Build.exe
• Windows build tool
• Orchestrator for building/traversing
the build graph
• Our version – 4.22.4093
• ‘dirs’ and ‘sources’
• i386 is default build target
Dirs
Dirs
Depth-first Traversal of Dirs
• SYNCHRONIZE_BLOCK/DRAIN
• Filter on platform tokens {i386},
{amd64}
Sources
Sources
Makefile.def
Makefile.def
MSBuild
WiX
Dirs
Sources
Sources
Makefile.def
Makefile.def
CMD
VC++

93. In-Market Branches

CoreXT - Build Contd.
PASS[0|1|2]
• PASS0 – Generation of files, copying of static content
• PASS1 – C++ compilation
• PASS2 – C++ linking, managed code builds, setup/packaging, binplacing
• PASS2_BINPLACE – binplacing of built bits (mostly from PASS2)
Project Inclusion
• Integrating your project into CoreXT’s build
• Dirs, sources, makefile, makefile.inc, placefile

94. Servicing Concepts

CoreXT - mydirs
• If a “mydirs” file is present in a
directory, build.exe will use it
instead of any “dirs” file to
determine what to traverse in
that directory.
• CoreXT build logic to
copy/rename “mydirs.<build
type>” to “mydirs” when <build
type> matches the current build
command.
Example: build layer
DIRS= \
Source \
<Root>
dirs
DIRS= \
Application \
Source
mydirs.layer
DIRS= \
Foo \
Bar \
Application
dirs
Foo
DIRS= \
Zoo \
Kernel
dirs
Bar
Zoo

95. CoreXT - Our Environment

CoreXT - Build types
Build
type
Command
Pre-reqs
Description
Retail
Kernel
build retail-kernel
N/A
i386 build of kernel, frameworks and setup
Retail
AMD64
build retail-amd64 –amd64
Retail
kernel
AMD64 build of AOS, client, select other kernel
components, and setup
Layer
build layer
.product:<layer product>
Retail
kernel,
AMD64
Build of the specified layer product on top of the
kernel
Retail
test
build retail-test [*]
Retail
kernel
Build of the test frameworks and projects. With ‘*’
builds extended set of test projects (BATs, TAEs)
Debug
build debug-buildtracker
N/A
Build of the kernel and frameworks in debug

96. CoreXT - MSBuild

CoreXT - Layer Build
• Builds the application component for a specific workload on a specific
layer
• Near 1:1 mapping of Product workload : Layer
• Exceptions – industry solutions in DAX6HF and DAX5SP1HF
• Settings for layer product: \build\productConfig.cmd
• Example: call %inetroot%\build\productConfig.cmd -Product
• Build command: build layer .product:<product>
• Example: build layer .product:FoundationSE
RUPayrollSE

97. CoreXT - Build

CoreXT - Layer Build Contd.
Prepare base environment
Import/Compile Product
Extract Result
Static*
CombineXPOs
UpgradePreProcessingAX*
SetupDatabase
GUIImportXPOs
SetupApplication
SetupKernel
GUICompileAll
PostBuildValidation
CreateBaseEnv(SE)
GUICompileIL
DTDrill
ApplySEUpdates
DeployLocalizedLabels
DeployApplication

98. CoreXT - Build Contd.

CoreXT - Layer Build Contd.
Step
Description
Static*
Unzips static content from RTM, CUs in checked-in zip files
SetupDatabase
Initialize the empty model and application database
instances
SetupKernel
Install the CD version of AOS and client
CreateBaseEnv(SE) Import any precompiled models that the layer product
we’re building depends on
ApplySEUpdates
Install any kernel hotfixes present on the CD

99. CoreXT - mydirs

CoreXT - Layer Build Contd.
Step
Description
CombineXPOs
Combine all XPO files for the layer product we're building into a single XPO
to import
GUIImportXPOs
Import the combined XPO file into the AOT
GUIImportUpgradeXPOs Same as GUIImportXPOs but for the layer product upgrade model if
defined
DeployLabels
Copy ALD (label) files to the AOS install directory, restart AOS, run
labelsync
Import VS/SSAS projects Imports and compiles VS and SSAS projects to the AOT
GUIComileAll
Compile the full AOT
GUICompileIL
Generate the IL assembly representing the AOT
DeployLocalizedLabels
Import the localized ALD files into the localized label models

100. CoreXT - Build types

CoreXT - Layer Build Contd.
Step
DeployApplication
UpgradePreProcessingAX*
SetupApplication
PostBuildValidation
DTDrill
Description
Take backup of DBs for bootstrap, export compiled
models for the layer product we're building
Copy the major version upgrade scripts to the CD
Copy application files to final setup location on the
drop (CD or SE Installer path)
Runs XPOVerifier against AOT to assert validity of
overlayering done
Export granular hotfix model(s) from the fully
compiled AOT using dependency data

101. CoreXT - Layer Build

CoreXT - VCSDef
• MorphX version control settings defined in dynamically-generated file
%inetroot%\Definition\VCSDef.xml
• Defines source/label file locations for the models in the current layer
• Layer template location:
%inetroot%\build\ax_public\deployment\tools\vcsdef_<layer>.xml
• Template values expand from branch and product configs (e.g. label
ID ranges)
• Changing layers: SetStartupLayer <new layer>

102. CoreXT - Layer Build Contd.

CoreXT - Label Files
• AX Label (.ald) files edited through the
label editor in MorphX.
• 1..* label files per layer product
• Located under
MorphX
User finishes editing labels in editor
and initiates submit
Label file locked
IDs allocated for new
labels
CoreXT
Sync/resolve label file
%inetroot%\source\application\labels
• LabelUtil command – validates
correctness of label files prior to
submitting
Run label file validator
Is label file
valid?
Yes
Direct submit label file
updates
No
Unlock label
file

103. CoreXT - Layer Build Contd.

CoreXT – Deploy.cmd
• Installs the AX components onto a
developer’s machine and initializes the
developer environment
• Discovery of latest official build
• Initialize test environment (import of
framework and tests)
• Used for local developer and gated
check-in deployments
• Source:
\build\ax_public\deployment\...
Discover/copy build
Execute AXSetup.exe
Import test framework/
code
Checklist items / init
environment

104. CoreXT - Layer Build Contd.

Gates

105. CoreXT - Layer Build Contd.

Gates - Execution
Machine allocation/
Preparation
(Sync, nuke, uninstall old deployments,
apply changelist)
Changelist and
checkin
validations
Build
Deploy
Servicing Tasks
Test
Post Steps
(kernel, app, test, debug flavors)
(enlistment deploy for
UTs and BP checks, WTT
setup for BVT)
(Collect dependency data
on change)
(BVTs, UTs, BP checks)
(Determine Results, Submit CL, Send report email, copy logs,
update bugs, teardown and release machine)
(DynCop, SDDump macro
validators, code coverage
pairing , etc.)
Test Leg Machine
Build Leg Machine
Return results
Retail-test
(core)
Primary Gate Machine
Schedule
Run
Preparation
Retail-test *
(optional)
Deploy
BP
Checks
Public
API
Checks
Copy core test back to primary
Pre Checkin
Retail-kernel
Retail-amd64
Layer
Deploy
Pre-Submit
SDDump
Changelist
Tests
Unit Tests
(C#, X++)
Determine
Results/Submit
Schedule parallel leg builds
Build Leg Machine
Debug-buildtracker
Return results
Post-Submit
SDDump
End

106. CoreXT - VCSDef

Gates - FastKernel
• Uses a pre-build kernel build to seed the gate process and skip the
retail-kernel and retail-amd64 builds
• Seed build share: \\dcsrdgtdfs01\KernelSeedDropShare
• Impact – can save over 60 minutes per checkin
• Invalidated by:
• One or more files not in whitelist:
build\wtt_checkin\FastBuild\FastKernelBuildWhitelist.txt
• One or more files in blacklist:
build\wtt_checkin\FastBuild\FastKernelBuildBlacklist.txt

107. CoreXT - Label Files

Gates - FastApp
• Uses a pre-build application drop to replace the layer build done prior
to PreSubmitSDDump
• Seed builds come from the Rolling Hotfix BuildTracker jobs
• Impact – can save over 3 hours per checkin
• Invalidated by:
• One or more files not in whitelist:
build\wtt_checkin\FastBuild\FastAppBuildWhitelist.txt
• One or more files in blacklist:
build\wtt_checkin\FastBuild\FastAppBuildBlacklist.txt

108. CoreXT – Deploy.cmd

Gates - SDDump
• Process used to capture the affected elements of a change
• Dependency data submitted to a central DB on DCSRDRPTSQLA
• Dependency Database per servicing branch: DependencyDB_<Branch>
• Data later used to generate a dependency bucket for hotfix packages
• SDDump Validation
• Macro (Pre-Checkin & Pre-Submit SDDump)
Changelist is unique
Changelist description contains bug ID
Bug is in the expected state and contains a valid KB number
Checkin layer is enabled for servicing
• Micro (Pre-Submit SDDump)
• Changes don’t result in deleted elements
• Changes don’t result in elements reverting to base layer logic

109. Gates

- SDDump Contd.
Prepare base environment
Prepare Dependency Tools
Apply Changes
Extract Result
Apply Changelist
Run SERestore.cmd
Import XPOs / VS Projects
Remove SDDump Triggers
Run SEReplace.cmd
Import SDDump Validators
CompileAll
Extract SE model
Create SE model
CompileIL
Run pre-extract
(validators)
Import SDDump triggers
DBSync
Save dependency data to
file
Baseline build/deploy
• Create scripts to
revert/restore nonX++ changes
• Prepare environment
without changes
applied
• Restore X++ changes
• Import dependency
toolset
• Prepare models
where changed
elements will go
• Apply changes to AX
environment
• Execute checklist
items for fully
built/synced AOT
• Disable dependency
collection
• Validate changed
elements follow
servicing rules
• Extract model and
dependency data

110. Gates - Execution

Gates - Multi-product Check-in
• Only in DAX6HF
• Checkin workflow to validate
changes to multiple industry
solutions in a single checkin
• Enables SYP and dependenent
overlayered chanes to IS
solutions to be submitted in
one change
• Primary machine does submit
and post-submit SDDump

111. Gates - FastKernel

BuildTracker Managed Builds

112. Gates - FastApp

BuildTracker Builds - Rollup
Purpose
• Produce a full kernel + application build to the tip of the current depot state
• Checkpoint build to execute automations against
• BVT, BAT, Full X++, Full Best Practice, LMC
• Refresh localization binaries that subsequent hotfix builds will use
• Build type for Cumulative Update releases
• Produces update package for all changes since RTM
Cadence
• Full Rollup master with validation occurs ~3 times per week
• Runs daily when stabilizing a CU release

113. Gates - SDDump

BuildTracker Builds - Application Hotfix
Purpose
• Produce a granular hotfix containing the X++ change fixing a specific product
issue + any dependent change since RTM
• Builds Application against RTM kernel to assert no dependency taken on
binary hotfixes
Cadence
• On-demand.
• Auto-initiated in R3 branches on successful check-in. Otherwise manually
initiated by issue owner.
• Also produced as part of Rolling Hotfix schedule

114. Gates - SDDump Contd.

BuildTracker Builds - Binary Hotfix
Purpose
• Produce a cumulative build of the kernel and frameworks components
• Create patch installers (.msp) to apply fix to existing deployments
Cadence
• On-demand.
• Also produced as part of Rolling Hotfix schedule

115. Gates - Multi-product Check-in

BuildTracker Builds - Jobs
Rollup Build
Binary Hotfix Build
Application Hotfix Build
Localization legs
Validate Parameters
Validate Parameters
Retail-kernel
Layer build
Retail-amd64
DTDrill
Code Sign CD1
Create Package
Patch
Validate Hotfix
Retail-kernel
Retail-amd64
Code Sign CD1
Translation-submit
Patch
Layer build
Retail-test-core
Retail-test-optional
Code Sign CD2
Create Package
Create Package

116. BuildTracker Managed Builds

UA Build

117. BuildTracker Builds - Rollup

UABuild – Overview (Core Concepts)
• UABuild – UABuild is our documentation build process
• Content is generated for word doc format, AX Help Server format, MSDN, and
Technet format.
• DXStudio – Content editor used by writers
• Content is store in DB: Axapta01 on DBServer: DCSRDUASQL01
• CoreXT/SD – Used for source control and to build content branch:
MainUA
• Get For Build – Process to pull content from DXStudio and check it
into Source Depot.

118. BuildTracker Builds - Application Hotfix

UABuild – Core Build Types
• A GFB (Get For Build) pulls content from a DxStudio project into
Source Depot
• HXS Build – Builds content into HxS format, and optionally publishes
the compiled content to MTPS (MSDN or TechNet) using DxPublisher
• HelpServer -- Builds content into a format suitable for the Dynamics
AX Help Server
• DocBuild – Builds content into Doc, Chm, and/or PDF formats.
• ManagedReference -- Builds MRef content into HxS format.

119. BuildTracker Builds - Binary Hotfix

UABuild - Core Build Flow
DXStudio
GFB – Based on
Project Name
Build is driven off
Environment Variables
for ProjectName,
Language, Forkname,
Target (Staging, Live).
Update Source
Depot Content
Folders For Project
UABUILD1-15
Build output to \\
dcsfrblddrop01\
MainUA\Build
\\dcsfrblddrop01\
MainUA\Build\
BaseDrop\
NoCheckin\EN-US\
Latest Content Pushed Here

120. BuildTracker Builds - Jobs

Daily Activities

121. UA Build

Assessing Official Build State
• Nightly builds scheduled, ran as expected, kicked off test runs
• Expected state - everything runs as scheduled and finishes in a timely manner
• Troubleshooting - For scheduling problems, check BuildTracker. For failures
and extended execution times, analyze the build logs.
• Evaluate build drop share resource check report
• Expected state - all drop shares report a healthy amount of free space
• Troubleshooting - if any drop shares show errors about low space (like the
one linked above)
• Run the bt.exe tool to get failed retention jobs and retry them. If they continue failing,
manually delete the drops
• If all retention jobs are passing, investigate if per-build size has increased or retention
policies have changed to result in more or larger builds on the drop. Log bugs accordingly

122. UABuild – Overview (Core Concepts)

Assessing BuddyBuild State
• Check on buddy build queue
• Expected state - majority of the time the queue is shrinking or 0 and jobs are
completing (queue can be growing during peak times)
• Troubleshooting - Check on machine availability (e.g. are any offline). Check if
running jobs are experiencing hangs or other systemic issues causing them to
take longer

123. UABuild – Core Build Types

Assessing Gated Check-in State
• Check queue length on MBSRPT
• Expected state - Queue as small as possible
• Troubleshooting
• Queue processor not running
• If queue is closed due to a break, work with owner of the issue to drive to resolution then
reopen the queue.
• Check if checkins are taking longer to complete or if there are fewer resources available to
schedule against
• Check running checkins
• Expected state - no checkins experiencing systemic issues resulting affecting the
execution of the gate workflow
• Troubleshooting – check WTT for machine states, check recent changes to
build\wtt_checkin

124. UABuild - Core Build Flow

Assessing Gated Check-in State contd.
• Look through latest failures
• Expected state - All failures have been classified with incidents, all non-dev induced
incidents have bugs
• Check machine pools
• Expected state - All machines in the gate WTT pools are heartbeating and are in a
ready or running state
• Troubleshooting
• If the machine is ping-able and you can connect, log on and try cycling the WTTSvc windows
service. You may need to change the machine status in WTT studio to Unsafe if it has been
without a heartbeat for an extended time
• If you can't connect, log an issue through http://dcsrequest for someone in the lab to take a
look
• In either case, if a checkin job was interrupted when the machine lost heartbeat, be sure to
inform the owner and resubmit their checkin/abort the old one.